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• ■ INTRODUCTION 

-J 

The need for many kinds of improvement in our educational system 
and its current practices is w]dely^ recognized in our Society. Attempts 
a^t Improvement, through applying scientific and technical advances have 
^fto^t^et brought about the changes in educational outcomes originally hoped 
for. Thq call for excellence in a system of democratic education 
(Gardner, 1961) has not yet resulted in^-marked alterations in the schools ' 

or ifi their manner of operation* The striking facts of turmoil in urban 

« 

centers have made even more apparent the failure, not of eKisting schools, 
but of our system of education^ to provide substantial segments of our 

population of young people with the skills they, need to become Integral 

. ^ ' • ' / . 
members of „ our society* *- / , ^ , 

' \ ^' / 

Research and research-related activit"tes ^development, diffusion, 

and Valuation) have an important part to play in efforts to effect 

changes in bur educational system for the pur|>ose of bringing aboit 

needed improvements. In many areas,^ is* systematic knowledge, 

including theory, that would apparently contribute most to such change— 

we simply do not know enough about causes of educational effectiveness 

to be able to take /Action. 1 In other inst^rit^es, it is clearly development 

that is most- needed--the new courses already developed scarcely make a 

I 

deht in the techniques 3nd procedures which will enable the schools to 

make optimal jise of new te^chnological advances* In many fields of . 

^ / * 

educational operation, it has become evident that evaluation is the 

activity most in need of emphasis— we have not yet fully learned how ^ 

to test the usefulness of new educational procedures or materials.. 



In stm other areas, it Is apparently dissemination of the outcomes* of 
reS^earcn and development which holds most promise~we know too Tittle 
as yet about how to bring about utilization of new ideas, techniques, 
and product^. ' . • 

The importance of research and research-related activities in ^ 
the field of edu^c^tion has become increasingly clear to many important 

' '4 I * 

segments of our society,-^ Indeed, the widespread acceptance of the 
criticality of these activities in building a khowledge basis thai^can 
be used to influence ^chool practiices ;has resulted in increasing^pubjic 
an^ financial' support for; research and research-related functions 
during, the past decade. \This increase in research, (development, 
diffusion and evaluation ende^ors has resulted in a parallel demand for 
qualified persons to participate in the conduct of these activities. 
Evidences of t/he need for suitably trained persons to perform essential 
functions in thp<;f^areas of effort^are many.* Public an4 fH^ivate 
sdiools. Research and Development Centers, Regional LabOra^ries, 
independent research agenSies, State Departments of Education, and univer 
sities and colleges have been hardV^essed during most of the period 
since 196S to find sufficient numbers of emnloyable persons qualified 
in requisite research, development, diffusvon and evaliiation skills. 
Based on an examination of persons employed on government projects and 
a p)f(ojection of expected funding, Clark ancf Hopkins (1969) concluded 
that these combined needs could readily create a discrepancy between 
the demand and supply of qualified jcfisearch and research-related 
personnel in the order of fpur thousand persons by 1974 unless more ^ 
effective ways are found to train persons to fill such roles. 



Since 19d8, deterioration on the economic scene and .failure 
of funding agencies to increase support far research and related activi- 
ties at projected rates have suggested that the research community may* 
j)0f^yffer from an acute shortage of personnel as soon as originally 
anticipated. In fact, the shortage, in temis of sheer numbers of 
vacancies unfilled, seems to have been temporarily alleviated by, a 
general depression in the job market. However, this "breathing spelt" 
has allowed a related problem to come into focu$ more clearly. It is 
\» becomingHjncceasingly apparent that educational research, development. 



diffusion, anid evaluation activities suffer from insufficient training 
of many role pccupants in these areas. It is clear that many persons ^ 
"performing inlresearch and research-related roles are severely handicapped 
by their lack of skilVartd/or knowledge in relation to substantive and 
^ methodological requisites * in their areas^ of investigation. It is 
/ equally clear that even ihe researchers larid, related personnel who are 



\ 



initially well-ty^aine*d 'to engage^in theijr V^ppctl^ activities quickly 

a^re faced wUh ^obsolescence in a jield e^^lmost devoid of viable programs 

desig>ied to keep such persons abreast of new developments in t^ieir 

area^ of specialization^ As Gage (1967) pointed out, there is a problem 

\of "obsolescence' among educational researchers resulting from the 

♦ 

many r^ew substantive and methodological develo^jnents in .the field, 
including -such technologies as Bayesi an Theory, computer simulation, 
organization theory, flexible scheduling, among inany others/ 

-in^an attempt to prevent obsolescence, the American Educational 

Research Association (AErI) has conducted during the past several years 

\ ^ / ' 

a numbjr of Pres^pssio/is of ^^eduaational^ research training, each, of 

several days' duration, ^pric^f to its AnniJal Meeting. ^ Several hundred 



\ 



\ 



ERIC 



i)ersbn$ have applied arfniially for admission into sessions dealing with 
topics such as experlmentaV^ design, .anthropological field methodology, 
ifiul tivari ate .design and ^DAlysjs, research management techniques, . 
instructional product development, evaluation,, noi?parametric statistics, 

'^d computer and, natural language.'- It is evident from^the number of 
applications that educational researchers themselves recognize their 
need for training in new methods and techniques, and that they are active 
in seeking such traiiiing in organized extra-university settings. The 

. proportior))^ needs satisfied by such sessions is unkn^n; but the size 
and scope of interest suggests a substantial unfulfilled de/mand. / ^ 



\ " "The AERA Presepsioris ha\/e been evaluated by 61ass.,( j.968) and 

~ ! ' ' ) '* ' ' i ' • . . 

Popham (T969).- Their findings 'emphasize the effect! venessi of training 
sessiois, even when relatively , brief , in raising the quality ofi^|rfor- ^ 
mance of educational researchers in specific areas of new methodology 
which are recognized as needed by the recipients. However, one'cannot - 
be certain from such evidence that all or even most of the needs tor 
postgraduate, upgrading, and contra-obsolescent skills are beirtg ade- 
quately met» • For one thing» the demands reflected by the topics of the 
^resessions may be inadequately reflecting needs ^of schools, distr;rtts, 
and state agenc^ies, as opposed to universities* This is in line with a 
suggestion stronc^y p^resented^^y Di Lorenzo (1965). And for another, 
the limitations n'n length of training provided by presessions may i 
preclude the offering of a number of kinds ^ training* which are of , 
' equal or grea^ importance, but which requfre more time. g 
Lack of knowledge aboEjt' "training variables" is undoubtedly the 
greatest impediment to planning training programs that will not^only 
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provide sufficient Irilti'al training- to researchers and .related personneV * 
but also 'provide sufficient Inservice train1ng-t*ff^f€vent obsolescence^ 
and continu'allj upgrade skills^. The problem may be/stated in brief 
terms~~as follows. Although the >ralue of .educational research .to the 
broader goal, of educational improvement is generally recQgn1z«d,"^.e. ' 
do not yet know precisely what Kinds of functions muf^t be '^^rformed by 
• the. educational research community in' progressing t. tms goal. As 
a corollary, we nave- not yet systematically defined jwhat kinds of, require* 

- — rments for educational research are generated by schools and other compo- ' 
nents of the educatioital system. The central question, arising from 
the existence *of such^ requirements, is: what are the skills of educ'a- 
tional researchers which will meet these needs? If^ a determfnatfon ^ 
can be made of these skills, the further step can presumably.be taken 
to discover jwhich of these skills are currently possessed by personnel' 
If) the field and whixh are miss^ingl This procedure will yield a list 
of deficiencies, or needed skills,. At sueh a potnt, systematic considera- 

, tion can be gnven lo~ttre~ question ^f how these needed skills can best 
be established: .Some of them may already.be the focus of emphasis of 
graduate training or ot^er programs': 'dther. needed ^.;ills can perhaps 
best be supplied by alternative inservipe approaches to "upgrading" and 

- "pr.eventing obsolescence" Which appear ^,ta^b6<tan itnportarit part of the 
total complex of educational research and re'search-related ti;fiining- ^ 
To begin to establish a relevant knowledge base and, develop 

' procedur^l^to attack these problems, AERA has established a Task Force 

on the Training of Research ai>d Resear(^«related Personnel in ^Education- 

^ ^ . \ 

Unde** support of the pres^« grants this first year, of Task Force opera- " 

tioa represents a pilot year in which several strategies for training. 
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research and research -related personnel and for coljecting'data related 
to training variables are being explored. The emphasis during the 



first year is not on the development of products or on the 



^nrfujCt of 

ly-controjled research Studies; rather. Task Force efforts haye 
^'oeen aimea at (a) gathering interim data relating to. current training 
needs and current training procedures in educational research and research- 
V^ned areas, '(b) s^timulating the development of quality/ins^lructional 
materials for use in preparing research and research-reVated personnel, 
and (c) planmng carefuUy control! e'd Idag^-range studies of manpower 
nee(is in educational research and research^related areas, competencies 
required of r/Ole occupants in such areas, ^and the effectiveness of 
existing and projected traiiling^ograms for preparing sufficient quanti- 
ties of well -prepared educational researchers and research-related 
personnel. . ' , 

^ Objectives 

! • - ^ 

The«objectives of the Task Force project for 1969-70 include, ' 

both short-term *anjl long-term; efforts. Specific objectives and'suby 

objectives of both types are listed belowj ' ^ ^ ^ / * 



/ 

/ 



— ^ ^ 

^Early in the project, the Task Force refine* afid articulated the 
original set of objectives, resisting in the present objectives- While 
different from the original set in wording and subordination, they do not 
represent substantive changes nor did they require procedural modifica- 
tions. The changes, were introduced to increase clarity and better 
represent the original intent of the Task Force in proposing the activities 
reported herein* ' I 



\ 



Long-term Objectives 

1. To plan long-range systematic studies to: 

(a) determine functions that research and research-related personnel 
\ , are required to perfomiy 

(b) determine skills that research and research-related personnel 
j need to perform such functions, 

(c}^* project new functions and skills that are likely to be required, 
(d) continuously monitor the relationship between skills needed > 

and skills produced by extant training programs in order to 

detect any discrepancies between the two. 

2; To plan how to develop specifications, models, and criteria for 
developing instructional' materials for use in training research 
and research-related personnel. 



Short-term Objectives 

1. To determine necessary functions for research and res^orc^r-related 
personnel--i .e. , to determine the nature of operattdnal requirements 
existing for such persons in various educational settings, 

2. To determine the skills that research and research-related persons 
nee^'to perform these functions, 

3. To determine which of the needed skills are adequately provided ' - 
(developed in sufficient^quantity) by current training procedures 
and which skills are not adequately provided by such training 

.procedures— i .e. i which skills are lacking. ' ^ ' 

4. To project new skills that will be needed in the future- 
s' To identiry alternative training modes and procedures that appear 

promising ror use in (a) upgrading present personnel in areas of 
lacking skills, and (b) preventing obsolescence. 

j5. To select two alternatives from among those identified as most likely 
to achieve the desired goals of upgrading and maintaining the pro- 
ficiency of research and research-related personnel, develop 
materials and procedures necessary for implementation of these 
modes of training, and conduct a tryout and evaluation of these two 
modes* 

7. To use data gathered from attainment of ^th^ above objectives to: ^ 

(a1 refine the plans for long-range studies focused on the topics 
encompassed in these objectives, and J 

(b) describe and make recommendations about a program for training 
research and research-related personnel that will be relevant to 
^ ' the desired goals of developing, upgrading a^d maintaining skills 
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■ • 

During the course of the project, progress toward attainment of . ^ 
the above objectives was reported in a series of technical papers 
produced by the Task Force,' These tGChnical papers represented interim 
reports and have been incorporated in edited form in this-report. To 
orient the reader to (a) the relationship between the objectives and ^ 
the remainder of this report, and (b) the relationship of the technical 
papers to the various sections of this report, the following discussion 
is included. In it, each objective is discussed and the relevant technicalj 
papers and sections of the present report are referenced. Short-term ! 
and long-term objectives are discussed separately. 



Short-term Objectives 

Objectives 1, 2, 3, and 4 are interrelated and can be considered^ 
as a groups. Procedures directly relevant to the objectives are discussed j 
herein in Chapter 2 (and, previously, in Technical Paper No. 1^, 2, 3^, andj 5) 
Briefly, this chapter contains a discussion^f (a) conceptual efforts to| • 
define relevant functions (e.g., research, diffusion) .and generate | 



lists of skills necessary to attain each function, (b) development and 
administration of an interview .technique to a representative sample of | 
employers of reserach and research -related personnel to determine 
which skiMs were most important, which in shortest supply, and what 
additional skills might be necessary in the/ future, and (c) reconceptudlli- 

zation of essential skills and knowledge. \(A1 though not prepared under' 

/ \ ' h 

support from the presertt grant. Technical r Paper No. 4_ included several' 

^ ; 

r 

I 

I . 



2 

A list Qf technical papers by number, author(s), and title if 
included in the list of references that appears latqr in this report,* 



n 

notions that influenced these further conceptual efforts reported, in . 
Technical Paper No. £ and in this report and was therefore distributed 
in the technical paper .series to reach a broad readership. Since justi- 
fi cation. for some positions taken in this report-are cor^ained in that 
paper, it is included as Appendix F herein. 

Chapter 3 and Appendix J also contain information that is relevant 
to objectives 2 and 3 in that specific skills required by employers -and 
possessed by applicants in the past three annual AERA employment services 
are discussed. These data were previously included in Technical Paper 

!• h % 19. » 19- * 

Procedures relevant to objective 5 ar§ reported in Chapter 5 
herein and in Technical Paper No. li and 12* Briefly, this chapter 
contains (a) a survey of other professional associations to identify 
promising alternative training modes and (b) a consideration pf . the use 
of simulation techniques to traia researchers, developers, diffusers, 
and evaluators, • 

During the course of the study, it became obvious that data on ' 

the quality and effectiveness of ESEA Title IV Graduate Research 

Training Programs were badly needed* Indeed, it was c.lear that no 

recent data about this program— obviously the major: current vehicle for 

training educational researchers—were available. Consequently, the 

Task Force formulated the following objective: 

To analyze characteristics of trainees in Title IV graduate 
Research Training Programs and characteristics of the programs 
themselves in order to provide needed baseline data. 

This objective was seen as a necessary first/step in a series of 

objectives aimed at collecting data about the competencies being taught 

/ 

in Title IV training programs, subsequent career involvement of Title IV 



graduates, etc. However, budgetary constraints prohibited the Task 
Force from pursuing even this one objecti.ye in addition to tiose to 
which it was .already cownitted. Therefore, the Task Force requested 
that they be allowed to substitute the "Title IV" objective above for 
objective 6 in the original list and expend project funds to attain the 
new objective. This request was approved. by the project officer in the 
National Center for Educational Research and Development in the U. *S. ' 
Office of Education and appropriate funds originally earmarked for 
Objective 6 were transferred to the collection and analysis of Title IV 
data^*. Procedures relevant to/this new objective are reported in Chapter 4 
herein and in Appendix L, which contains codifig formats for all trainee 
data analyzed. Interim reports of these activities appeared earlier in • 
Technical Paper No^. 1_3 and 14. 

Although objective 6 was removed as a formal requirement under 
terms of the grant, the Task Force had already stimulated the development 
and evaluation of two training activities that' are (at the. time of this 
writing) underway. Specifically, one training idea generated from within' 
the Task Force seemed to have sufficient merit to warrant conducting 
a tVyout^and evaluatiorl and was submitted as a separate proposal* 
Another proposal for developing and evaluating research training 
materials was cosubmitted by a Task Force member. Although the actual 
conduct of these projects will not appear in this final report, the 
original ir,,ent of objective 6 seems to h^ave been at least partially 
fulfilled through the Task 'Force role in stimulating these activities. 
These activities are discussed in greater detail in one section of 
Chapter 5 and Appendix M herein and, earlier, in Technical Paper No. 12. 
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In objective 7, attainment of subQbjective 7a is implicit in 
the plans for long-range studies presented in Appendix N of this report. 
Although not discussed explicitly., all data reported >n this report have 
infiluenced the plann/ng of long -range studies proposed in the continua- 



\Hion proposal. Indeed,' many of the short-term objectives and Task 
Force activities during the pilot year existed solely to probe Tiew data 
sources and data collection procedures to identify sources and procedures 

that hold promise for more extensive long-range stutty. The attainment 

\ * } 

0^ subobjective 7b is at least partially represented by. this final report. 
Although the pilot year activities have yielded mpre new questions thai? 
answers to old questions, some characteristics of ^sirable training 
progra,ms have been described and interim recommendations made'herein. 

Long-term Objectives * 

Objective 1 in this category has been attained; the proposal for continua- 
'^I'on which is included here as Appendix N contains a brtef description 
bf long-range systematic studies des;igned to collect all the da^a specified 
in, that objective* More detailed plans for the studies are on record 
In Task Force files.. ' ' 

I Data presented in Chapter 3 and Appendix J herein also represent 
an initial step toward establishing the monitoring system called for 
in subobjective Id. Thesip data earlier appeared in Technical Paper 
No. 6, 7, 8, 9, 10, and^li. 

. Long-range objective 2 has been only partially attained. The 
Task Force was at one point reftjuested to suggest specifications and 
criteria for materials developmentx^'-oposals for consideration by the 
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U. S. Office of Education Research Training Branch, Although the 
response to this request might technically satisfy the grant requirement, 
the Task Force views it as only a partially satisfactory response and 
efforts to develop more adequate specifications, models, and criteria 
for such endeavors are currently underway. 

Overview of thi:& -Report 

The remainder of this report , is organized inta (a) four chapters 
in which Task Force activities, data, and results are reported, 
(b) a sutimary and conclusions chapter, and (c) 14 supporting appendices. 
It should be stressed that this is a report oPfesearch and devel opmerrtal 
efforts £e]r se and, xonsequently, the conclusions and reconwiendations 
.included herein are* restricted to those thai^^^be inferred rather 
directly from the data and procedures discussed in this report. Task 
* Force deliberations have resulted, in addition, in a number of positions 
and recc^Twendatiions , Indirectly related to but not a direct outgrowth 
of the content of this report. These position statements and recommenda- 
tions that are based more on collective judgements than on data will 
be presented later in a separate document ( Technical Paper No . 18 , * 
to be distributied in February of 1971). 




CHAPTER 2 

A CLASSIFICATION SYSTEM FOR RESEARCH-RELATED SKILLS 

i 
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A CLASSIFICATION SYSTEM FOR RESEARCH -RELATED SKILLS 

, . . „ " :> 

One of the most serious impediments to efforts to plan training 
programs for research-related peV-sonnel is lack of knowledge about which 
particular competencies or "skills are most important in, conducting research 
and research-related activities. In this chapter, the effoVts of the Task 
Force to obtain such knowledge are reported. 

The discussion is in three parts. The development of a classifi- 
cation system for functions and skills required of ^research-relat^ 
personnel is presented in the first section. In the second section, t*he 
development of a method for obtaining information on certain competencies 
isT described and the data collected by that method are presente;d. A ten- 
tative proposal for refinement and reconsideration of the essential know- 
ledge and skills for educational research and evaluation is presented in 
the third section. 

The Dev^opment of a Classification System for Functions 
and Skills in Educational Research* ' . 

This section contains a discussion of the ijnitial proced^jres used 
in developing the classification system for research and research-related 
functions and skills. The procedures reported herein included the follbwing 

(a) preliminary Task Force discussions of the parameters within which 
research and research -related functions could be described, the intferrela- 
tionships among those functions, and skills relevant to each function, 

(b) synthesis of these concepts into a working draft, and (c) further Task 
Force discussions to react to and refine the draft into a tentative position 
paper. 



/ ■ 

Introduction ^ ' 

Preliminary discussions resulted in several decisions and products. 
First, it was decided that the' focus should be broadened from research, 
ger se, to iTiclude all phenomena related to inquiry in educat'ion. Thus, 
whether or not an activity could be shown to be directly related to inquiry 
became the major criterion ^for determining whether it should be considered^ 
further. Secondly, sevewl functions relevant to this focus were discussed 
and attempts made to interrelate them'. For functions on which there was 
consensus (research, development, and diffusion), tentative liSts^of skills 
were drawn up* 

The next step was to draft a synthesis of the conceptual efforts, 
ilicluding efforts to delineate the area of evaluation. The result was a 
cUssification system for functions and skills required of researTch-related 
personnel in education. This system was modified, definitions all 
functions were added, and additional skills were identified for inclusion. 
What follows is this preliminary attempt of the Tssk Force to develop a 
classification scheme for research-related' functions and skills that could 
be examined and reacted to by practitioners in these research-related 
functions J 



This presentation is in the form in which it^as originally distri- 
buted to research-related practitioners. Although the necessity of some 
content revisions was apparent, they were deferred so that the procedures 
.for interviewing the practitioners and the results of the **realit^ testing" 
reported in later sections of this chapter could be interpretecl in relation 
to the content actually reviewed by the sample of practitioners. 
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Functions Required of Research and Research-related ftersonnel , 

Research 
2. Research-based Development 

a. invention and engineenitg 
„b, product testing 
Diffusion 

a. . dissermnation 

b. demonstration 

2 

c. facilitbting adoption ^ 

3 ' 

4, Evaluation 



context evaluation/situation analysis 



b. program plannin^g/input analysis ^ 

c. process evaluation/program monitoring 

d. outcome evaluation 

Definitions a,hd lists of skills for each function appear on the 
following pages. 
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These functions are largely undefined at present and, pending further 
development of roles anil skills in these areas, the Task- Force focus wjll 
be on those functions that can be confidently and directly related to dis- 
ciplined processes of /inquiry. In addition to such inquiry skills, the 
functions referenced here also, depend on ski.lls and knowledge, that might 
more appropriately be developed in fields not directly involved in producing 
research or research^related personnel per se (e.g/, communications theory, 
marketing, engineej^/ingl and / consequentlyT training in these non-inquiry 
Skills might be viewed as a rather minor coopera^tive trafning responsibility 
For AERA, with mofe attention directed to more directly relevant inquiry skills 

3 ' 

The evaltfation functions that evolved from Task Force procedures 
parallel closely types of ^valuation previously described by Daniel L, 
Stuff lebeam, and some terms in 4 a, b, and c above are borrowed or adapted 
from his work. 



Research is the activity *aimed at obtaining knowledge, either 

general or specific. This knowledge may be obtained by empirical or other 

\ , * 

systematic methodis and it may or may not have immediate application. This , 
knowledge may resuUj'n theoretical models, functional relationships , or ^ 
descriptions (such. as the amount of teacher-talk occurring in a\^ecif1c 
clas^oom). • ' ' ^ " * f 

The process by whiph such Hcnowl edge is produced generally involves: 
(1) the specification of a question to be answered or a hypothesis to be 
tested; (2) the delineation a population of interest and a sample of 
that population; (3) either the description of experiences. .shared by or 
*the administration of a treatment to the sample; (4) measurement (including 
observation and evjdence gathering); (5) analysis of evidence; and (6) ''\ 
drawing conclusions and implications, 

The rajor' distinction between research and research-related ^unctionj 
*(such as research -based development and evaluation) is that in research, 
utilization of the knawledge is typically not foreseen in the same specific 

> 

detail as it is in research -related activities. . <. 

t 

Example , Several techniqtf^s of sequencing problems and 
generalisation^ in presenting mathematical concepts are compared in terms 
of their effect^ on initial learning, retention, and transfer of the concepts, 
in 'order to studythe relationships between sequencing in task presentation 
and valued learning outcomes. 
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, ' Skilll Necessary for Research ' \ 

!• Drawing research implications from* results of priot' research , 
studies, . • , ' 

2, Identifyin'g Snd delineating, significant 'researchabie pr6bl,er?«. 

3, Procuring and/or managing respurces (material and human) 
necessary to recfch research objectives; ^. , 

4, Interpreting, evaluating, and synthesizing relevant literature* 

5. Formulating hypotheses or empirical que$ti(*s to be answered > 
by the^^tudy. ^ ^ - ' ' 

6, Specifying data or. evidence necessary for a rigorous test ' ^ 
of th^ hypothesis. ^ ' " \ * 

7. Identifying the population to which results should be generalized \.| 
and a sample representative of that population, using appropriate \^ 

* sampling techniques to draw the sample, ' 

8. FormtH'atlng alternative generalizations from predicted research' • 
outcomes. * 

9. Identifying appropriate research' methods. ^ , ^ 

10- Uhderstanding experimental quasi -experlmen^l , ^and other 

systematic approaches to inquiry* and drawing on such know- '\ 
* ledge in designing a research study appropriate to the probjem 
under consideration. r 

4 

r 

11. Applying the reseav^ch design, recognizing, explicating and 

controlling threats to validity. . ' 

1?. Identifying classes, of behavioral outcomes for measurement. 

13. Choosing specific variables and treatments (where appropriate) " 
to be \ised. . ^ ' . 

I ^ 

14. Selecting appropriate techniques of measurement. - • 

15. Developing measuring instruments • ^ ' ■ 

16. Assessing the validity of outcome measures. 

17. Using. a variety of data-gathering methods (tests, interviews, ^ r 
analysis of documents, etc,^. 

18* Organizing data for analysis. - 
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19. tinderstanding the general role, tyDeSj and assumptions underly- 
ing various statistical techniques, and drawing on such knowledge 

i in seleiting and Using appropria.te, techniques of data analysis. 

20. Using aids in data analyses, such as computer-. processing. 

'21, ' Inteipprl^ttng i cl c awing appropriate conclusi^s m& 
implications from. data analyses. : . 

22. Formulating statenjents' of a theory that offers an explanation 
{c4Mse-ef feet relationship) of the behavior unden t tudy- 

23. ReporjbiRg research findings and impli editions, orally and in 
^ writing. ' . ' * 



fhial form with very casual or no, field testing (e.g.,. 



Resea.rch-ba.sed development is th"at activity which uses the results 

o'f systematic, disKipliped inquiry to lead'to'the cpeation of an educatijona 

" ^ ' ■' ' , 

product (e\g.-, instructional materials, grading system for a school ,ide8ign 

for sefiool b'uilding). Hiis is in contrasj;- to development activities which 

have been |)rou9hV to 

^ teacher-prepared instructional unit), x 

Example . The Pittsbu^rgh R and D/Center^ developed feVning 

c materials f^r Individually Prescribed Instruction. The materials were 

developed 'to meet pejfdrmance specifications and were subjected to several 

revisions as a^-tonsequence of field tests with the intended audiencps until 

the performance specifications were attained. * . 

Product testing^ is that aspect of research-based development which 

is focused on actual testing of. products that are viewed as ready for 

tryout. Product testing involves. the collection aad analysis of data to 

assess discrepancies between 'product outcomes flmd objectives. It may take 

place either in th'e laboratory or ir fiel(|^settings »and j's used repeate^dly 

during the developmental phase "as the product^ is^^^ther refined or modified 

' on the basis of performance on previous product tests. - * 
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Example . Sections of a proposed 'progranmed mathematics text, 
based on principles of operant conditioning, are Evaluated by. panels of 
experts. After the sections are revised to incorporate the experts' 
suggestions, small samples of students at the level for vWiich the text. 

is aimed use and evaluate these sections. After further revisions 

I 

prompted by difficulties encountered by the students, the sections are 
combined and the total text is tested on larger samples in real school 
situations representative of the audience for whom the text is designed. 

Skills Necessary for Research-based Development 
(including Product Testing) 

1. Interpreting information concerrning education goals. 

2* Drawing on research results in planning developmental 
activities. , 

! 

3, Cdnceptualizing^systems, their elements, and interrelations 
among these elements* 

4. Specifying. desired perfo^nce joutcomes (objectives) of 
instruction, / 

5: Devising techniques to/f3efmfy entry capabilities of 
learners, • ^ 

6. Identifying alternative instructional and media techniques. 

7. Determining appropriate sequences of topics 'in instruction. 

8. Describing the product to^be developed. 

9. Composing effective, oral and written forms of instructional 
communications. 

10. Directing the work' of production personnel. 

IK Selecting or devising appropfiate techniques for tneasurihg 
•outcomes. i 
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12. Designing and managing initial labOr^htorii^^ests of developed 
.techniques and materials, • 




13. Designing and managing field tryouts and tests 
14* Reporting evaluation of dtucomes. 

15. Interpreting evaluation findirlgs. ' ^ 

16. Specifying requirements for revision based upon outcome 
evaluations. r ' , 

*' Oif fusion encompasses planning, designing, and conducting 
activities which insu>^ the application 4n educational programs of the 
findings or products of research and development efforts. This may be 
dene by various means, including (a) the use of communication techniques 
to disseminate information about the product or findings, (b) the -conduct 
of demonstrations to establish the utility and applicability of the product 
or findings, and (c) procedures which^ facil i tate adoption or application of 
the product or findings. 

Example > In junior-high' science teaching, inquiry methods 
have been developed into methods of question-asking by teachers. It is 
-now necessary to (a) inform appropriate persons^ and agencies about these 
methods, (b) demonstrate their utility, and (c) provide help to those who 
wish to adopt the methods (e.g. , provide training in use of the quest4;op- 
asking techniques to accompany the introduction of a new course in sc4pnce 
in a particular junior-high schpol). * . 

* 

Skills Necessary for Diffusion 
Dissemination ^ ^ / 

V 

1. Defining and analyzing characteristics of target group(s). 

2. Selecting from all available information about developed 

, packages that which jian be most effectively disseminated. 
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Selecting the most* effective dissemination vehicles tp convey 
information to target gro\jps. 

Composing the information, within a chosen format, for accurate 
and pervasive disseminat'ion. 

Implementing actual dissemination, including the^directlon of 
technical production personnel . * 



6, Designing and implementing techniques for evaluating the effec- 
tiveness of the dissemination effort. 

Demonstration ^ 

1. Specifying" nature of the demonstration, 

2. Selecting appropriate setting and personnel for dei^onstration. 

3. Managing and coordinating the demonstration effort, 

4. Evaluating the effectivenebs of the^ demonstration 
Facilitating Adoption 



1 
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Identifying features of tha adopting organization Qr system 
which differ from those in which the product was developed * 
and tested- ) ' , . 

2. Designing nriodifi cations' of the product to fit the adopting 
organization or system," when necessary, | ^ 

/ ' ^ ( ' 

Designing, procedures for modifying the adopting, 'system or 

/-^organization to fit the product, when necessary, including 

tJhe design of needed training programs, • 

4, Identifying potential barriers to implementation. 

5, Devising and conducting long-range evaluation pf the installed 
package. 

Context evaluation/situation analysis is the process of identify- 
ing and comparing intended outcomes of a system (what is ,'desired) with 
actual outcomes (what is) on specified variables in order to (1) identify^ 
needs and problems to which the system must attend, and j,2) provide irv^rma- 
tion which will help decision-makers to 'develop relevant objectives which, 
if attained, will satisfy the need or solve the problem} " . 

V ■ . ■ : ' . 



Example , tomparison of a current reading" achievement level 
of elementary school children in District X j^^ith national norms on the 
same reading test leads to identifica0t)n of a discrejDancy of one full 
grade level- Data collected on^possible causes reveal that the teachers 
feel inadequate in teaching Vending, hence they spend as little time as 
possible on it. An elective is set to develop an inservice training 

program to develop teacher competency in teaching reading. 

/ - 
/ , , 

/ 

^Skills Necessary for Context Evaluation/Situations Analysis 



^/ K Identifying goals of the system. 

2. Assessing the social relevance of those goals, 



3. Identifying values that are implicit in the system goals. 

4. Identifying the nature of the standards or rtorms the decision- 
makers will apply in interpreting the relevant data which 

may be provided. . ^ ^ 

5. Clarifying and explicating desired outcomes of the system. 

6» Measuring current actua-1 outcomes of the system through 
technique? such as: 

' at demographic analysis 
, b. economic analysis 

c. psychometric analysis 
systems analysts 

e. observat;ional techoiques 

7. Comparing actual and intended system outcomes to identify 
discrepancies (needs) which exist in the system. ' 

8. Explicating the proi)lems that create the needs and diagnosing 
the causes of these problems. 

9. N 4ieTping system personnel, to develop objectives which, if 

attained, will satisfy the needs or solve the problems iden- 
tified above. 

^10. Designing a monitoring system that will 'provide continual data 
(of the type above) on the status of the operating system. 
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Program planning/input analysis is the process of identifying and ^ 
assessing the pbtentiaf utility o- alternative approaches designed to ' \ 
attain specified objectives. Once objectives have been established, the 
research-related aspects of program planning/input' analj^sis are: (1) 
Identifying what needs to be done to attain those objectives; (2) IdePti- 
fying the financial, political, and personnel limitations that Impjlnge upon 
attainment of the objectives j (3) identifying alternative approaches and/or 
materials for use in attaining the pbjectives; (4) determining the financial, 
political, ahd personnel cost?, for each alternative and the degree to which 
it contributes to the attainment of^he objectives; and (5) assignirrg 
relative weights to each of the objectives, ^ ' 

Example , A decision has been made that the reading instruc- 
tion program in grades K-1 is not satisfactory and a new or modified 
program must go into effect in one year. Data collection about four 
alternative programs reveals that one is superior t6 the others on a 
majority of relevant criteria. The steps listed above were necessary in 
reaching this conclusion. However, these activities do not include the 
final choice of the program to be implemented. 

Skills Necessary for Program Planning/Input Analysis 

1. Helping system personnel to apply criteria to lists of possible 
objectives in* order to select those which are feasible within 
constraints of the operating context, 

^. Helping system personnel to establish priorities for the 
selected objectives, 

3. Identifying and rating alternative strategies for attaining 
, . the selected objectives, " 
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4. Identifying and rating available resources (human, material, 
and financial) and/or potential sources of support. 

5. Selecting a strategy for implementation. 

6. Selecting a source of support or the available resources which 
will be used to implement the program. 

7. Predicting the potential barriers to success in the proposed 
course of action and judging the potential of the strategy 
for overcoming the estimated procedural barriers. 

8. Identifying alternative tactics tp implement selected strategy 
'and choose those that seem most likely to succeed. 



Process evaluatiort/p' r ograaifi monitoring' is the process of (1) 
monitoring an installed program to detect unanticipated problems or devia- 
tions'^from design or specified procedures and (2) providing immediate feed- 
back, to program operators for their possible use in making program 
irodifi cations., ^ 

Example . A new reading program scheduled to begfh on Septem- 
ber 1. requires that overhead slides be used extensively durin^ the first 
three weeks of instruction to show specific wdrdls and pictures. Program 
monitors might inform the program operator that (ia) during the firs.t week 
of instruction, the projectors were used by only 7 of 16 teachers in the 
school and (b) this low rate of usage is apparently due to difficulties 
th^ teachers have encountered in operating a new, more complex overhead 
projector. 

* 

ar 

Skills Necessary for Process Evaluation/Program Monitoring 

1, Designing and selecting indicators of progress in educational 
programs. 
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2. Monitoring the program to detect deviations from design or 

^ specif-^d procedures through techniques such as unobtrusive 
measures, systems analysis, and observational techniques* 

3, * Anticipating predicted barriers and remaining alert to 

^ unanticipated problems that threaten the success of the^ 
program- 

4» Providing immediate feedback to program operators for their* 
possible use in making decisions about modifications of the 
plan, procedures, or resource alloations. 

5/ Perceiving human relation problems that threaten the success 
of the program. 



Outcome evaluation encorn)asses the identification, collection, and 
presentation of information useful to those wrio must, judge the worth of ^ 
educational program, product, or procedure, typically at their^teftni nation. 
The data to be collected might include (a) intermediate or terminal evidence 
on the attainment of objectives, (b). unanticipated problems and beneficial 
side effects, and (c) costs in both human and material resources. Such 
infOHMtion would be used by the decision-maker to determine whether the 
program or procedure should be continued as is, modified, or terminated* 

Example * A team of school -district evaluation personnel 
design and conduct an Experimental investigation to test the relative 
effectiveness of two competing sets of reading materials in developing 
specified reading skills in a particular group- of disadvantaged children- 



Skills Necessary for Outcome Evaluation 

1. Applying appropriate designs to evaluation studies. 

2. Developing general criteria and designing data collection 
procedures for application in measuring the effectiveness and 
efficiency of existing innovative practices and products, 
i*e., minimum standards and outcomes which indicate successful 
utilization of practices and products. ^ 
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3, If necessary, translating objectives into behafvioral terms, 

A. Identifying situations In which the designated behavior can be 
observed and recorded. 
K 

5. Establishing standards or norms for judging whether objectives 
^ have been attained. 



6. 

7. 

8. 
9. 
10. 

n. 

12. 
13. 
•14. 
15. 



Selecting (or developing) and using techniques of measurement 
to yield Information relevant- to these standards* 

Assessing the validity of outcome measures. 

Collecting and organizing the data preparatory w analysis. 



Selecting an appropriate technique to analyze the data. 



Analyzing the evidence yielded by the evaluation, 



1 



Judging the* strengths and weaknesses of the pi ans 1 and ^pro-* 
cedures erapi^oyed for meeting the project objectives* 

Deciding how to explain the outcome as a function pf plans, 
procecJyreS t and resources* 

Deciding what recomnendations to make as a result ^f the 
outcomes. 

Estimating the potential imf^ak:t of the outcomes on the problem 
area being ser^ved,. 

Providing sufficient information to the decision-maker to enable 
him to decide whether to continue, modify, or termipate the 
activity or process evalioted* 



16* Specifying changes that need to be made in the context evalua 



tion system due to^ decisions about program continua 



:ion. 
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The Development of an Interview Technique and 
Analysis of Interview Results 

Following development of the initial classification schema reported 
above, it was distributed to a sanyle of persons involved in research- 
related activities in order to elicit their reactions to It through personal 
interviews. A second purgose for conducting the interviews was to collects 
interim data on manpower deficiencies and methods for alleviating such 

needs, A description of the procedures for designing the Interview tech- 

•/' 

nique a_nd Identifying the intervieiwees Is contained in the first part of 
this section. The results of the interviews are reported in the second 
part. ' > . 

/ 

Development of the Interview Technique 

The procedures were developed by the Task Force and refined and 
detailed by the project staff. In general, they included the following 
steps: (a) identifying an /appropriate sample of inWviewees to react to 
the draft of the classification schemgi and respond to critical xjuestioris 
formqlated by the Task Force, (b) developing and testing an interview 
schedule for use in eliciting and recording 1nter\?iev>ee reactions, (c) i 
developing ajj^^stin^g a technique for distributing the classification 
schema to inlerview€;es and arranging to conduct the Interviews, (d) 
identifying and training Interviewers, and (e) distributing the classifi- 
cation shcema and conducting the Interviews. Each^of these steps is 
discussed below. 
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The Jnterviewees • 

' There were fOur major considerations in determining how the inter- 
view^els would be identified. First, it was apparent that the most relevant 
interiiewees would be persons who administered agencies in which research 
and research -related activities were conducted and thus employed or super-j) 
vised others who participated in these activities. Second, it was 
obviously necessary to include injthe sample some interviewees^ with major 
responsibility for supervising or employing researchers, some with major 
responsibility for evaluation, etc. Third, an attempt was made to include 
interviewees from each of ten institutional settings^identified as those 
in which educational research and research -related activities are con- 
ducted. Fourth, 60 was set as., a minimal number of interviewees necessary 
for collecting Ihe needed information, / 

Tlie ten institutional settings included the following: universities 
and colleges', research and developinent centers, regional educational 
laboratories, independent research agencies, state departments of educa- 
tion, scjjool districts, Federal agencies, military services, business and 
industry, -and professional education associations. A matrix was"^ formed^ 
with these ten settings on one dimension and four major functions— -research, 
development, diffusion, and evaluation— on the other, yielding 40 cells in 
the matrix. 

The Task Force nominated as interviewees persons with whose profes- 
sional responsibilities they were sufficiently acquainted to be certain 
they met the. criteria of (a) employing or supervising persons engaged in 
one or more of the four major functions, and (b) being sufficiently 
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apainted with the function(s) to know what competencies are necessary 
iri its (their) performance. -The functional area or areas for. which .each- 
nolninee had responsibi.lity and expertise Was also identified. 

Eighty-two persons were nominated as interviewees. Using their 
functional area(s} and institutiona1*setting. the. 60 persons who distri- - 
buted most equally across cells ^ere chosen as interviewees. In setting ' 
up interview appointments, it was found that eight of the interviewees , ^ 
were ill, out of the country." or otherwise unavailable*.* "Eight^new inter- 
viewees were selected as replacements. The resultant 60 persons who 
served as interviewees^ were distributed by rows, columns.^and cells ; ' 
as shown in Table 2.1. . ^ . ' 



%preciatidn is expressed to th€ following interviewees for th* 
' L^rarBa?r?'f J u^i^rT '° '"I'l 'I'^^'i ^^-^-n^Alkin. Alexander Astln. 
■yicX cVeutkt CVa f Tf''^"^"?!: °' Walter Borg. Lee 6. Burchinal. > 

Vianf plhoJj i Z i,^' Clemens. Lewis Crum. Gabriel Della- 

Pjana. Robert A. Dentler. Richard A. Dershimer, John Easter. John C 
Flanagan Warren G. Findley. Robert B. Glaser. Gene V/Glass Keith Goldhanmer- 
R ^9°" e- Guba. Robert L. HammondrThomas Hastings . * 

R chard Hills. Paul Hood. Kenneth D. Hopkins. James Jacobs. Herbert 

RloerTenn'on^'plJnh'T'''^''' ^r?PP' Roben Lankton. 

2S2!^H S ^^^f,^ Susan Markle.-Ward^'Mason. Donald M. Medley. 

S?^?nl"?'''''"""'A"?^°^¥'!^^^' ''^^"'^l^" W- Neff. Roland Pellegrin. 
S^mJ n ^Tf '/r^''? '^1^" Robinson. James Robinson. 'Wade M 

HaJrC ? '.n^''* Scanlon. Charles Schapp. Richard E. Schutz. Harry Shoemaker. 
Harry F Silberman. Robert Stake. Theodore 'R. StorlW Richard Turner 
Risden Weston, Asahel Woodruff. Lome Woolatt. Lou1s%ne. and Jall^s'young. 
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Distribution of Interviewees by Institutional Setting 
and Functional Emphases: Sampling Plan 

m- * 

* " V 
% ■ " ■ ■' ' L — J - ■ I ■ ■ I p 



• ' Numbers of Interviewees with Functional Emphasis on ; 



Institutional 

Setting Research 


Development 


Diffusion 


Evaluation 


Total 


Universities and , 
Colleges 


12 


8 


• 5 




17 


Research & Development 
Centers " 

Regional Educational^ 
Laboratories 


6 

3 


4 

4 ..^ 


4 


2 
4 


7 * 

♦ 

7 


Independent Research 
Agencies 


,4 


3 ^ 




, 3 - 


5 ■ 


State Departments of 
Educati on 


1 


2 ' : 


3 


3 


4 


School Distrl^ats 


4 


2 


4 


7 


8»' 


Federal Agencies 
Military Services 


2 

.2. 


, . 1 
1 


.2 

0 » 


1 \ 


4 ■ 
2 


Business and^ Industry 


3 


3 


. 2 


2 


3 


Professional Educjeitional 
Associations 


, 2, 


-0 


1 


■;ol ^ 


3 


. Total Interviewees 


'39 


28 


% 24 




60 



^OTE: In many instances, an interviewee was listed in more tharj on6 cell \ 
. in the same row* Therefore, row totals are,not sums of cell totals^ but ^ 

represent the total number of interviewees within each institutional setting. 

Column totals represent the total number of persons across all institutional.; 

settings interviewed in relation to each function. 



Of *the 60 interviewees, it. was found that 54 were either present 
or past participants'^mselves in the tunctfbns for which they were 
selected. In fact, a perusal of^ the names of Interviewees Msted in 
footnote 1 shows that many of the interviewees are among, the leading 
experts in educational research and research-related activities, V - 

The 60 interviewees were drawn from a broad spectrum of insjl^ltu* 
tions. These institu;tions and agencies are Tisted. below. Numbers In 
parentheses ^re use^d in instances where more than oh^ interviewee was 
drawn from an institution, ' ' ' 



Universities 

Indiana University (3) 
Pennsylvania State University . 
Unfversity of Colorado (3) 
Bucknell University ^ 
tJhip State University (2) ' 
Syracuse University 
University of Utah (2) 
Florida State University 
University of Illinois (3) 

Regional Laboratories 

Far West Laboratory for Educational 
Research and Developm&fit (2) 

Southwest Regional Laboratory for 
Educational Research and Develop- 
ment 

Mid Continent Regional Educational 
J Laboratory 

Central Midwestern Regional 
Educational Laboratory . 
Center for Urban Education ^ 
.Research for Better Schools, Inc, 



R_& D Centers 



Learning R & li Center, Pittsburgh (2), 
Wisconsin R & t) Center on Cognitive 

, Learning o . *' 

R&D (Center in Educational Stimu- 
lation, Athens, Georgia 
Center for Advanced Study of 

Education Administration, Eugene, 
, . Oregon (2) ■ . 
R&D Center for the Study of 

Evaluation of Instructional - 
- Programs, Los Angeles ' 

Independent Research Agencies 

American Institutes for Research (2) 
Educational Testing Service * 
Educational Research Council of 

America i 
Institute of Educational Research 

State Education Departments 

Illinois State 6epartment of Education 
New York State Department of Education 
Pennsylvania State Deparipept of 
'Education 



School Districts 

Detroit Public Schools 
Pittsburgh Public Schools 
Cincinnati Public Schools 
Kearn, School District, California 
Sequoia Union* Hi^gh School District, 

California 
George Washington High School,' 

Alexandria, Virginia 
New York Publje Schools 
Col umbus^ Public. Schools, Ohio * 

Federal Ag^ie^ 

U; S. Office of EdWtion (4) 



Military Services 
' USAF Academy 

, USAF Besearch Training Coni(jand. 

Business and Industry 

Systems Development Corporation 
American Telephone's Telegraph' 
. Harcourt, Brace & World 

Professional Education Associations 

American Council on Education 
American Educational Research- Assoc, 
National'^Education Association 



^ Instrument DeveVopment ^ % ^ ^ ' 

An interview .schedule was developed, critiqued and revised. The 
revised schedule was- tried out with three educational researchers and 
revised again fo correct ambiguities and problems identified-. The result 
^as a highly structured inter\^iew schedule constructea so as .to prdvide 
spa^ie for recording responses^directly on ;the instrument. The final tnter- 



view schedule is reprotli^ced in full in, Appendix A. 
Contacts with Interviewees 



"Was no'l-ftconomi call y feasible to interview e,ach member of the 
sample personally. Therefore, .it was necessary to develop a technique 
for conducting telephone interviews id all but a few cases where proximity 
allowed personal' interviews to be conducted. To facilitate the conduct 
of the telephone interviews,,^ technique *was developed .where the conceptual 

sche^na Wcts sent to each interviewee, with a letter explaining, the project 

• ' / ^ . ' 

and the necessity of contacting him to get s reactions to specific parts 

of the schema^^ A return postcarcPwas included for the interviewee's use 
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ijjj^indil eating times when he could be reached by telephone or in person, by 

the interviewer'. This procedure was also tested with the same thre,e 

educational researcl\ers as part of the instrument tryout. The cover 

letter and postcard were then revised; the final form of each is also 

shown in Appendix A. • . / 

W 

Prior to distributing the materials to interviewees, the name of r 
the' person serving as 'interviewer was recorded on both the letter and th^ 
return postcard, which was also addressed to\retur^directly to that inter- 
viewer.. Instructions oil these processes also appear in Appendix A.- After 
materials were distributed and return pos+cards received, appointments 
were made (using jqng-distance appointment operators where long-distance 
inter'views were to be conducted). 

The Interviewees 

y 

Seven Task Force members, the project director, and three project ' 
staff members served as interviewers^ Thirty-four irtterviews were' cor^- 
ducted by Task Force members and 26^ by project staff. Because of (a) 
the involvement of the Task Force and the present staff in devel opting 
and revising the instrument schedule and (b) the highly structured nature 
of the interview schedule, little training was necessary for these persons 



_ >: r 

5 / ^ \ * 

Appreciatipn is expressed to the following members of the Task 
Force and fellows in the Laboratory of Educational Research, University 
of Colorado, for their assistante to the authors in conducting the 
telephone interviews reported herein: Nancy W. Burton, Abbot L, Ferriss, 
Robert M. Gagne", William J. Gephart, John E. Hopkins, Jason Millman, 
Susan J. Ol-defendt, W. James Popham, Ernst Z. Rothkopf, and James R. Sanders. 



to conduct interviews^ JJone^eless, all interviewers were thoroughly 
trainCT ifl-all relevant concepts and techniques reiquisite to qjbnducting- 
the interviews. Instructions iwere given to all interviewers in an attempt 
to standardize tb£ way in which the interviews were conducted. 

Conducting- the Interviews 

The, interviews wefe conducted between November 15, 1969- and March 
18, 1970. Interview^ were completed with all 60 interviewees. In addition 
to responding to questions, the interviewees made suggestions for moc^ifying 
tho conceptual schema and lists of skills. These suggestions and the ^ 
data analyses were used in subsequent attempts to modify and improve 
the conceptual schema tested ^herein^ 

Analysis of Interview Results 

Interviews were held with 60 persons who either employed or super- 
vised research or research-related personnel in one pf^ ten types of 
institutional settings.' The results of analyses of data collected during 
these interviews are contained in this section. ' 

Analytic Framework and Techniques * 5 

The primary framework within which the data were analyzed consists 
of two dimensions, insti tutionaFsetting (10 levels) and function (,7 levels) 
Ju)^taposi tion of these two dimensions results in 70 cell combinations 
within which the data were analyzed (as shown later in Table 2.2) in rela- 
tion to elicited responses. The analytic techniques- consist solely of 
simple descriptive techniques, such as frequency distributions , averages 
and percentages. 
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Results . Each afrllysis of data reported/fn this section is , 
preceded by the item in the Interview schedule that was used to collect 
the data* The items are sojjDetimes abridged or modified here for clarity 
in presentation,^ / ) 

Necessary FUncti-ons^ ^ * * 

1. / Please look at p/(ge 3 of the working paper that was sent to 
/you. (See page^ig above) Whifch of these functions do you 
as necessiiry in your program or institution? (.Refer 
/ THuerviewee to definitions and examples itnecessar^J 

/ The responses of the interviewees to this [item are surmnarized 



in Table Z.zJ 



g 

^ i Items appear in tfieir original form in Appendix A. 

•t , ♦ 

^Cell entries in Table 2.2 reflect the number of persons w each 
institutional setting actually interviewed in relation to each function 
and include four changes from the earlier presen*tation of the number of 
persons within each institutional setting interviewed in relation to each 
function. (See Table 2.1) First, the interview schedule ftovided for 
data collection in relation to seven functions, rather than the four listed 
in' the previous table. (Because of apparent ."differences in the four 
activities listed under "evaluation," each was treated as a separate func- 
tion in the interview schedule, thus increasing the ^number of functions r 
considered to seven.) Second, two persons tiad moved from the institution 
specified for them in the earlier sample plan and, consequently, they were 
interviewed in relation to their new responsibilities as employers of « 
research-related personnel. Therefore, there was a slight shift from the 
number of interviewees listed in the, sampling plan for each institutional 
setting and the actJal number of persons interviewed in each setting. ^ 
Third, comparable cell totals in the two tables'differ since in the earlier 
one they represent a priori expectations of functions in which the employers 
• supervised employees , wherea^^cel 1 totals in Table 2.2 herefn' represent 
the functions in which employers were found actually to be supervising 
employees. Fourth, two interviewees responded incorrectly or inadequately 
to the questions and their responses were deleted from the analyses; 58 
responses are included in all analyses in this report. 
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Table 2.2 

Number of Interviewees, Listing Each Function as Necessary in Their 
Program: By Institutional Setting 



o o 



ERIC 



Institutional u o- -r- ' ^ tJ 8?ti 

Setting S (u 2*- Hrz 



<u 



O'- Q OUi »— icC D-LU OUI f— 



University 16 15 7 15 12 14 14 19 

Regional Laboratory 2 , 5 3 4 -* 3 3 3 6 

R&D Center- '5 7 4 4 3 3 ,3 7 

Indep. Res. Organization 3 5 4 43 3 3 5 

State Dept. of Education 2 ^, 1 1^" 0 2 1 2 3 

* School District 554 '7 6677 

"Federal Agency 1 1 2 2,2 2^1 3 

Military 2 1 1 1 1 1 1. 2 

Industry 2 2 ' 3 1 1 1 2 3 

Prof. Educ. Assoc. '' 2 0 3 3 2 1 1 3 

Total Interviewees 40 42' 34 41 35 35 37 58 

1^^ _J . . 

Note: In many instances. an interviewee was listed in more than one cell in the same 
row.l Therefore, row totals are not sums of cell totals, but repres'ent the 
total! nn{riber of interviewees withtn each institutional setting. Column total* 
represent^the total number of persons across all institutional settings inter- 
viewed in relation to each function. 

* ' i 

, When the four functions relating to evaluation are considered together, 

47 of the 58 interviewees listed at least one of the four as necessary in 
their program. Forty-two interviewees listed deveTopment as necessary, 40 
listed research, and 34 listed diffusion. These totals represent "absoHite** 
necessity with no consideration of which function (when multiple functions 
were listed as necessai^) was most essential. An analysis of the 'Velative" 
necessity of functions appears in the^^next section. 
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Priorities Among Functions ^ 

.The interviewees were asked to indicate the relative importance 

of the functions they lasted as necessary in thei r- programs . 

a. Please rank the functions you have mentioned in the order 
/■of their importance in attaining the goals of your program. 

It was difficult to devise any meaningful statistics for use in 

reporting the results of this part of the interview since the nunter of 

functions being ranked varied from one interviewee to the next. For 

example, a rank of "3" given by one interviewee who listed only three 

, functions relevant to his program could not be assigned the same meaning 



as a rank of "3" given by an interviewee who li stfe^s^en<functions. 



Therefore, the rankings of functions have been presented in three 
different ways for the reader's information. 

First, the ranks assigned to each function (irrespective of the 
number of functions rapked) were averaged across institutional settings 

o 

for each function. The results are presented in Table 2.3. 

Second, the frequency with which interviewees listed a function as 

being thfe most important, by institution, is provided in Table 2.4. The 
^meaning of this table is straightforward and the data may be compared 

across institutions. 
-- * JMcd>Jt)ie_functions were dichotomized on the basis of their 

rankings into primary and secondary functions. If a function 



If it could be assumed that there were no systematic differences 
across institutions in the number of -functions listed per interviewee, 
the mean rank of- ■a function could be used comparatively across institu- 
tions; unfortunately, it would be dangerous to make such an assumption 
with the small number of persons interviewed and the obvious differences 
in numbers of functions ranked. Therefore, data presented in Table 2.3 
should be interpreted with caution. 
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was ranked in the first half of the total number of functions, it was 
considered to be of primary importance. If the function was ranked in 
the second half of the total, it was considered to be of secondary 

Q 

importance* The resulting frequencies are presented in Table 2*5. 



Table 2.3 



Average Rank of Functions in Order of Importance in 
Attaining Program Goals: By Institutional Setting 



Institutional 


\ 


C • 


c 


§ 


< 




C 

o 


Setting 


Research 


ivelopn 


o 

lA 
3 
«*-. 


-*-> 

>^ 


vt 

■M >. 


</) <a 


g3 








3i— 

a. lO 


o « 
> 


•«-> 10 




& 


a 




c c 


O. UJ 


3 > 
O Ui 


University 


3.25 


2.47 


4.14 


2.71 


2.54 


2.93 


3.33 


Regional Lab. 


2.50 


1.20 


1.67 


3.25 


3.00 


2,67 


2.00 


R&D Center 


2.00 


1.00 


4.25 


4.50 


4.00 


3.50 


3.33 


Indep. Res. Organ. 


1.33 


2.00 


4.25 


2.50 


2.00 


2.00 


1.67 


State Educ. Dept* 


2.00 


6.00 


2,00 • 




1.50 


2.00 


1.50 


School District 


4.60 


- 5.80 


3.50 


2.29 


1.71 


1.83 


1.86 


Federal Agency 


1.00 


1.00 


2.50 


1.00 


1.00 


1.00 


1.00 


Mi 1 i tary 


3.50 


7.00 


5.00 


1.00 


2.00 


4.00 


3.00 


Industry 


4 00 


3.00 


3.33 


1.00 


2.00 


3.00 


4.00 


Prof -Ed. Assoc. 


-1.00 




2^.-33 


r.33- 


— 1~50 


'"J. Op 


"X^O " 


Average 


2.90 


2.59 


3.38 


2.57 


2.27 


2.55 


2.68 



There is some question about how 'meaningful these data are, since 
the ccitegorizatlons are, again, correlated with the number of functions 
listed by the interviewee. 
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Table 2.4 ' 



/ 



Frequency of Listing of Each Function as the^Most Important 
Function in Attaining Program Goals: By Institutional Setting 



Institutional 
Setting 



o 



to 



O O 



c 
o 

4J -P 

01 3 
4-> 

C «o 

O > 
O UJ 



to 

CL «d 

£5 




University 


7 


7 


1 


4 


2 


3 


3 


Regional Laboratory 


- 0 


4 


1 


0 


0 


0 


1 


R&D Center 


3 


7 


1 


0 


0 


0 


0 


Indep. Research Organ. 




3 


1 


2 


2 


2 


2 


State Dept. of Educ- 


\; 


0 


.0 


0 


1 


0 - 


1 


School District 




0 


3 


2 


2 




3 


^ Federal Agency 




1 


. 1 


■' 2 


2 


2 


1 


Military 




0 


0 


, 1 


0 


0 


0 


Industry 




1 


1 


1 


0 


0 


0 


Prof. Educ. Assoc. 


2 


'6 ' 


^ 1 


i 2 


1 


;0 


0 


Total 


19 


23 


10 


13 


10 


9 


11 



Note : TJie nuj*jr^ functims Jj5ted as jnost ijnpontant-exceeds-the 
nuiSber of interviewees because of. numerous instances where 
interviewees insisted upon assigned tied ranks of "1" to 
multiple functions. - 
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' Table 2.5 



Frequency of Lifting Function as Primary or Secondary in Attaining 
Program Goals: By Institutional Setting ^ 



InstUutioiial 
Setting' 



Si 

m 
(/> 
cc 



c 

i 

CL 

o 

01 

> 

0) 



o 



01 3 
o > 



I 

CO 4-> 
</> <Q 
0> 3 

o 5 
$- > 
a.uj 



11 

±1 > 

O LU 





P 


S 


P 


S 


P 


S 

— /■ — 


P 


S 


P 


S 


p 


s 


P 


S 


University 


8 


8 


9 


6 


3 


4 


9 


6 


'8 


4 


■ T" 

8 


6 


5 


9 


Regional Laboratory 


1 


1 


4 


1 


1 


2 


• 0 


4 


0. 


3 


0 


3 


1 


2 


R & 0 Center 


4 


1 


7 


0 


1 


3 


0 


4 


1 


2 


1 


1 


1 


2 


Indep. Res. Organization 


2 


1 


4 


1 


1 


3 


2 


2 


2 


1 


2 


1 


2 


1 


State Dept. of Educ. 




1 


0 


1 


1 


2 


0 


0 


2 


0 


1 


0 


2 


0 


School District 




4 


0 


5 


3 


3 


5 


2 


6 


1 


5 


1 




2 


Federal Agency 




0 


1 


0 


1 


1 


2 


0 


2 


0 


2 


0 


1 


0 


Military * 




1 


0 


1 


0 


1 


1 


0 


1 


0 


0 


1 


1 


• 0 


Industry 




1 


1 


1 


1 


2 


1 


0 


1 


0 


1 


0 


0 


.1 


P^pf. Educ. Assoc. 


._2„ 


._o_ 


Q. 




-1 


2 ... 


__2 - 


_]- . 


._?_ 




. . o_ 




0 


2 


Total " ' 


22 


( 

18 


26 


16 


13 


23 


22 


19 


25 


11 


20« 


14 


18 


♦ 

19 



P « .primary, S » secondary 
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The data presented in Tables 2.3, 2.4, and 2.5 are difficult to 
summarize. Wheli the average rank assigned to functions is considered, 
three of the four evaluaHon functions were ranked higher (considering 
"1" as the highest rank) than the other activities, followed in order by 
development, outcome evaluation, research and diffusion. However, when 
the frequency with which functions were listed as the most important is 
considered,' the order of importance becomes development, research, context 
» evaluation, product evaluation, diffusion and input analysis (tied), 
and process evaluation. When functions are dichotomized into primary. and 
secondary importance, development is most frequently listed as a primary 
function, followed by input analysis,, research and context evaluation (tied), 
process evaluation, product evaluation, and'diffusion. If the four 
functions related to evaluation are considered collectively, evaluation 
emerges clearly as the function most often listed as most important in 
attaining the goals of..the respective programs. Development is next 
most Important, followed closely by research, and diffusion is the least 
important function in most of the agencies represented in the interview 
sample. Di-f fusion was most often listed as a function of secondary impor- 
tance. 

•■ 

" MditTohTI'TTeseanbTi^^ ~ ~ ^ - . ^ 

Each interviewee was also asked the follo^^ing question: 
I ^ ^ S ir 

* Are there other research or research-related functions neces'sary 
either now or in the. future in the conduct of your program that 
are not included in out list? (If yes) Could you describe 
them (it) and give an example? 

No interviewees responded "yes" to this questonj'no additional ' 
research-related functions were Identified.- 



ERIC 



46 



1 



Research and Re search -related Skills 

The remainder of 6ach inter^view focused prfmarily on eliciting . 
information about skills^^ necessary to perform the functions the inter- 
viewee had listed as relevant to his program, A series of five questions^ 
wUh additional subdivisions, was presented to each interviewee for each 
of the functions he listed as relevw£~(e.g. , if the iriterviewee listed 
all seven functions as relevant, 'he wouTd be presented with seven seriLsi 
each of which contained the five basic questions, modified to refer to 
the respective functions.) * ^ 

Some of the questions were focused directly on information about 
skills necessary to'^rform the relevant research and research-related 
functions. Other questions were somewhat ancillary and analyses of ^ 
responses to such questions are contained in apjsendices. The five 
basic questions are presented below, along with an Indication of which 
analyses are Summarized in the body and in each appendix.^ ^ 

2. How many people do you employ or supervise who engage 
in research? 

3. Looking at the list of skills on page 5 (see page 21 above), a 
which of them do you consider the most important or critical 
to the performance of research in your organization? Please 

f ♦ identify rio more than five or ten. (List below their numbers 

, from page 5.) - 



^ In retrospect, the term "skills" may be less descriptive of the - 
items listed in the first part of this chapter than other terms such as 
"tasks" or Vcompetencies .". The extent to which kncJWledge and applicational 
abilities ai\e intermingled in these "skills" is unclear, but it is obvious 
that they differ in specificity and clarity. They are considered here as 
tentative l,ists that will need further evaluation and empirical testing 
by the Task Force and project staff. 

^Hhe exw^e given here is for research, but the questions were 
compa'^able for each of the other functions. 
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a. Are there any of these skills which you consider unim- 
portant in performing this function in your setting? 
(List by number.) 

• • # 

# b. Are there other skills which are not includedin the 

list on page 5, but should be added? 

4. Referring to those skills ypu have identified as necessjary to 
f ;. engage in research, which of them are "hard to come by"? 

, In other words, in *rhich of the skills that are necessary dtf 
you find deficiencies in your present personnST or in personnel 1 
you have attempted io hire? Please identify those that you 
feel are most inadequate amorig your present personnel or most - 
difficult to obtain through hiring new personnel. (List by 
number.) \ 

5. In relation to these Iskills you have just identified as being 
in short supply, Niow difficult do you believe they are to 
develop? For example^ could they be developed in inservtce 
training programs or .would long-range training be necessary 
(e.g., acafdemic year f^nstitute or graduate prog||ims)? » 

a. ^ Do you know of any existing training programs which 

are designed to develop the?e skills in trainees? ' 

b. Do you know of anywhere outside of existing training 
programs when;* these sitills are being developed inciden- 
tally to other activities? \ 

c. Can you suggest any new techniques or\ methods for train- 
:: ing personnel in these skills^? \ 

6. Are there skills that are not now necessary but whfch you think 
will be necessary to engage in research as ybur program con- • 
tinues, develops, or assumes other functions in the future? 

Question 2 was asked as a check to make certain that the interviewee 
in fact supervised or employed research or research-related personnel' and 
was an appropriate person to tntervtew. The average number qf ^sersons 
supervised or employed by each Ifiterviewee is shown J n Table 2.6. 

Responses to question 3b are summarized in Appendix B and responses 
to question 6 are summarized in Appendix C- Responses to questions 5, ^ i 
5a, 5b, and 5c are summarized in Appendix D. 

Responses to the questons of , most critical concern-.-3, 3a, and 4-- ■ 
are.SAinmarized for each function in the remainder of this section... 
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Research . ' ^ < . 

Forty interviewees listed the research function as being relevant 
to their program. These interviewees were asked to list the most ijnpoftarvt 

or cril/ical skills for the performance of research in their organ nation 

\ " • ^ 12 

from the list of skills provtded.tljem/ The resulting frequencies are 

shown in Table 2.7. The total frequency for each skill and the percent 

of respondents listing each skill critical areVesented in the kst - 

\ ' • ' 

two columns in the table. ' '" ' 

An arbitrary crit^V.ion for the 'practical significance of irtpor- 
tance for. each skill was set at 50 percent' of the interviewees listing ' 
thfr skill. Thus research skills 2, 9, 10, 11, 15, 19, 21 and 23' (see 
foldout Appendix E.) may be considered td be comparatively more' important ' 
or m6re critical skills 'than tfie other skills listed, ifsVindicatftd by • 
the interviewees. * 

^ The^telephone interviewees were asked which skills they considered 
to b6 unimportant in performing the research functidn. The results of 
their responses are given in Table 2.8. ' - 

It is evident that there was general agreement that none of the 
skills could be identified as being, unimportant. ' , . 

Another concern of the Task t&rce wa^^rto identify, from among the 
'Skills listed as important for each functtdn',/thbse^that are in short 
supp^ly- The MntervieweesT were . asked to consider' those research skills, 

12 • • ' ^ ' 

Th^s list of skill^and. the lists for the other functions are 

presented in Appendix E, which can be folded out for the reader^s con- 
venierxe in interpreting the skills that ^re listed by^number* through- 
out thirs section. 5 



Table 2.7 ' 

Frequency of Listing Skills as Most Impdrtant or Critical to tne 
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Note: n - the total number of i 
who responded to the 



ndividuals from each institutional setting 
list of sk*ills for this function. 
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Table 2.8 



Frequency of Listing Skills as Unimportant to the Performance 
of Research:. By Institutional Setting 
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Note: n = the 
who 



total number of individuals from eacl;y institutional setting 
responded to the list: of skills for this function. 
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which they had listed as being critical or important in their program 
and to indicate which of those important skills are "hard to come by"— 
i.e., in whi<h of them did the employers find deficiencies in their present 
personnel or in personnel they attempted to hire- The frequencies of inter 
^igVfees listing each research skill as iniportant, but^in short supply, are 
given in Table 2.9 • ' 

An arbitrary criterion of 25 percent agreement among all inter- 
viewees that a skill is important and in short supply was set for deter- 
mining the pratical significance of the listed percentages. Thus, 
research skills 2, 10, 15, 21, 22 and 23 were considered to be bqlth 
important and in short supply. 

^ ^Sevelopment 

Forty-two interviewees listed the development function as being 
relevant to their organization. < 

the most important skills for the developrr^nt function, as seen by 
intervieWees, are indicated in Table 2.10. Using again the arbitrary 
criterion of^Ox^ercent agreement among those interviewees, responding to 
a given skill, four skills were identified as being most important: 3, 4, 
11, and 13.\In terms of unimportant skills, only skill No. 7 was iden»^ 
■ tified as bein\j relatively unimportant by a sizeable number ofHnter- 
viewees, as shjowrtxin Table 2.11. 

There were=-three skills identified as being highly important 

, development skills that are in short supply--skills 3, 4, and 11. The 

criterion used for identifying these Skills was 25 percent agreement among' 

interviewees. The frequencies of response by interviewees are shown in 

\ • 
Table 2.12. \ • • . 



Table 2.9 

Frequency of Listing Important Research Skills in Short 
Supply: By Institutional Setting 
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Note: n = the total number of individuals from each institutional setting who 
responded to the list of skills for this function. 
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Table 2.10 

f "" " ' * 

Frequency of Listing Skills as Most Important or Critical to the 
Performance of Development: By Inst1tulflona1 Setting 
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Note: 



n = the 
who 



total number of individuals from each institutional setting 
responded to the list of skills for this function. 
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Table 2.11 

Frequency of Listing Skills as Unimportant to the 
Performance of Deve1opn)entt By ^istitutional Setting 
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Note: n = the total number of individuals from each institutional setting 
who responded to the list of s^kills for this function. 
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Table 2.12 



Frequency of Listing Important Development Skills 
in Short Supply: By Institutional Setting 
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Note: n = the total nui*er of individuals from each institutional setting 
who responded to the list of .skills for this function. 
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Diffusion 

Thirty-four interviewees listed diffusion as a fuoction relevant 
to their program. ' ^ 

The msi^ important skills for engaging in the diffusion function, 
using the 50 percent agreement criterion,^ were identified as skills 5, 4, 
and 6 under the heading "dissemination," skill 4 under "demonstration," 
and skills 4 and 5 under "facilitating adoption." The frequencies of 
response for the most critical skills are given in Table 2.13. 

Few skills were identified as being unimportant. It is evident 
from the data^resented in Table' 2.14^, however, that thf^ interviewees 
tended to consider most of the dissemination skills to be relatively less 
important than those listed under demonstration and facilitating adoption. 

Of those skills identifi^ as being important ski 1 Is, three were 
considered to be in short sqpply: skills 4 and 6 under dissemination and 
skill 5 under facil'itating ado'ption. The frequencies with which inter- 
viewees identified skills as being important and in short supply are 
given in Table 2.15. The 25 percent agreement criterion 'was used to 

identify the^^e three skills.. 

> 

Co ntext Evaluation/Sitii&tions Analysis 

Forty-one interviewees listed context evaluation/situations analysi 
as a function relevant to their program. 

Five skills were identified (using the 50 percent agreement cri- 

♦ 

terion) as being the most important skills needed for .performing the 
context evaluation/situations analysis function: skills 1, 5, 6, 7, and 

10. Responses on this item are summarized In Table/2^.16. 
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Table 2.13 



Frequency of Listing Skills as Most Important or Critical to 
Performance of Diffusion: By 'institutional Setting 
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Note: n = the total number of Individuals from each institutional 

setting who responded to the list of skills for this function. 
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Table 2.14 

Frequency of Listing Skills as Unim|«frtant to the performance 
of Diffusion: By Institutional Setting 
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Note: n = the total number of individuals from each institutional setting 
who responded to the list of skills for this function. 
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-Table 2.15 



Frequency of Msting Important Diffusion Skills In Short 
Supply: By Instttuttonal Setting 
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of individuals from each institutional setting 
the list of skills for this function."- 



ERIC 



Table 2.16 

Krequency of Listing Skills as Most important or Critical 
to the Performance of Context Eveluatlon/Situations 
Analysis: By Insitutional Setting 
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Note: n = the total number of indivictuals from each institutional setting 
who responded 'to the list of skills for this function. 
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rt'is evident frdm the data 



\in Jal}l 



e 2.17 that fe^ interviewees 



considered any of the listed context avaluation/situations analysis skills 
to be unimportant . 



T^ble 2J7 

Frequency of Listing Skcills as Unimportant to the 
Performance of Context Evaluation/Situations 
Analysis: By Institutional Setting 
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Note: n = the total nunter of indivicfuals from each institutional setting 
' who responded td the list of skills for this 'function. 
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.Of the skills identified as being important, most were found to be 

.in s>iort supply, according to the^ interviewees.* Using the 25 percent 

* 

agreement criterion for purposes of isolating critical skills* in' short 
supply, skills 1, 2, 3, 5, 6, 7, 8, 9, and 10- were identified. Responses 
to this item are sunmarized in Table 2.18, j . 

^ ' Table 2J8 " \ 

Frequency of Listing Important Context Evaluation/Situations 
Analysis Skills in Short Supply: By 'institutional Settin^y ^ 
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Note: n = the total number of individuals from each institutional setting 
who responded to the list of skills for this f unci ton. . ' 
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\ 

Program P1anninc}/Input Analysis 

* - - Y ' 

Thirty-five i^ntervieweej) natfied progi^arti planning/input analysib as 
a function relevant' to their program. 

Almost all of the skills listed for program planning/input analysis 

r 

were identified as being important. or critical to the performance of this 
functien. The frequenetes tjf responses by^ tTlstitution are^hown in / 
Table Using the 50 percent criterion^ skilils 1, 2, 3, 4, 5,7 

* ^ . X . Table 2,19 ' . 1 . ' 



.Frequency of Listing Skills as Most Important or Critical to 
the Performance of Program Planning/Input 
^- " An$ilysis: By Institutional Setting 
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vNote: n = the total number of i/idiyi duals from each inStitutional^setting 
who responded to the list of skills for this function'. 
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and 8 were tonsidered to be the most important -skills. No skills were 
considered to be unimportant, as shown in Table 2.20, although skill 6 

^ i 4 

was identified more frequently than the others in^he list. 



Table 2.20 



Frequency of Listing Skills, as Unimportant to *the Performance of 
Program Planning/Input Analy^sis: By Institutional Setting 
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Note:, n = the total number of individuals from each institutional setting who 
responded to the list of skills for this functionr. ^ 
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Of those skins considered to be important, three--$kills 3, 5 
and 7--were identified most frequently to be in short supply, using the 
25 percent agreement criterion. The frequencies of responses for each 
skill are presented in Table 2.21. 

^ Table 2.21 

Frequency of Listing Important Program Planning/Input Analysis 
Skills in Short Supply: By Institutional Setting 
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Note: n = the total number of individuals from each institutional setting 
who responded to the list df skills for this function. 
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Process Evaluation/Program Monitoring 



Thirty-five interviewees listed zhe pj^cess evaluation/program 
monitoring function as relevant to their o/ogram. 

Of the five skills listed for process evaluation, each one was 

identified as being an important skilf by more than 50 percent of the 

// 

interviewees. The frequencies of responses for each skill are given in ' 

Table 2.22. / 

I 

None of the five skills y/as identified by more than two persons as 



being unimportant, as can be seen in Table 2.23 




Four of the five skills identifijgjcUa^HjfThg important were listed 
as being in short supply by 25 percent or more of the interviewees. The 
frequencies of responses for those important skills in short supply are 
provided in^Table 2.24. 



Table 2.22 



Frequency of Listing Skills as Most Important or Critical to the 
.Performance of Process Evaluation/Program Monitoring: 
by Institutional Setting 
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Table k.23 



Frequency of Listing Skills as Unimportant to the Performance of Process 
Evaluation/Program Monitoring; By Institutional Setting 
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Table 2..24 • ' 

Frequency of Listing Important Process Evaluation/Program Monitoring Skills 
in Short Supply: By Institutional Setting 



Institutional Setting 
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the total number of individuals from each 
institutional setting who responded to the 
list of skills for this function 



Outcome Evaluation 
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Thirty-seven, interviewee's listed the outcome evalualtion function* 
as relevant to their program^ 

Seven skills were^ identified by 50 percent or more of the inter- 
viewees as important skills for performing tfie outcome evaluation function. 
These skills are 1, 2, 3, 5, 6, 13 and 15 on the list. Responses for this 
item are sunmarized in Table 2,25. 

It is evident from the data in Table 2.26 that few skills were 
listed by the interviewees as being unimportant. 

Of those skills which were considered to be important, 25 percent 
or more of the interviewees listed the following skills as in short supply: 
1, 2, 3, 6,12, 13, 14 and 15. The frequencies of responses for each 
skill are given in Table 2.27. 

Discussion 

It is difficult to separate for discussion functions which are as 
closely related as the seven functions considered here. Many of the skills 
listed under one function could easily have been included under another, 
and in some cases doubtlessly have been, using different terminology. For 
this reason it should be recognized that discussing the functions as 
separate, independent activities ignores interrelationships and overlaps 
among them. This difficulty aside, some results deserve elaboration. 

It is evident from Tables 2.2 and 2.3 that persons located in each 
of the institutional settings ^listed may engage in a wide spectrum of 
research-related functions. Although no surprise, this may serve as a 
caution against the common fallacy of stereotyping a position by the type 



Table 2. 2-5 . 

Frequency of Listing Skills as Most In^ortant or Critical to the 
Performance of Outcome Evaluation: By Institutional Setting 
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Note: n = the total number of individuals from each Institutional setting 
who responded to the list of skills for this function. 
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Table 2.26 



Frequency of Listing Skills as Unimportant to the Performance of 
Outcome Evaluation: By Institutional Setting 
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the total number of individuals from each institutional setting who 
responded to the list of skills for this function. 
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Table 2.27 . 

Frequency of Listing Important Outcome" Evaluation Skills in Short 
Supply: By Inistltutlonal Setting 
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Note: n = the total number of individuals from each Institutional setting 
who responded to the list of skills for this function. 
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of institution in which it is located. If the sample chosen by the Task 
Force is representative, it appears that all research or research-related 
acti^vities may have a home in any type of institution. The priorities 
plac|d on the functions may differ from one type of institution to 
another (Tables 2.3, 2.4 /and 2.5) but it remains that all the research* 
and research.-related functions listed in the first section of this chapter 
are relevant to programs in any of the institutional settings used herein. 

The data are clear in s' pportin^ the importcince of all seven func- 
tions suggested to the inter<^iewees* In terms'^of relative importance, it 
appears that evaluation, development and research rank in that order but 
all are high and close together on the scales used. Conversely, diffusion 
is viewed as relatively less important by ^ majority of the interviewees.' 
Although this may be partially attributed to the fact that fewer ''d^ffusers" 
were included in the interview sample than persons >vith functiorj^emphases 
in, other categories, it is doubtful that this alone would account for the 
size of the diVcrepancy. Perhaps the proliferation of roles for diff users 
embodied in current literature on educational change is prophetic rather 
than descriptive of present. professional priorities. 

It appears that the Task Force was most successful in identifying the 
inportant skills for the performance of the context evaluation and, process 
evaluation functions. All of the skills listed for these two functions were 
considered by the telephone interviewees to be important or critical skills. 
However, the percents of responses for the importance of each skill for 
these two functions may be somewhat inflated since the lists were compara- 
tively short, resulting in few (or no) forced choices similar to those 
required In the ranking of research skills. It remains, however, that the 



interviewees identified each of the "skills on the^list quite frequently ' 
*as leing ilra)ortant. 

The interviewees identified as one of the most important develop- 
ment skills "selecting or devising appropriate techniques for measuring, 
outcomes." This skill is one that is common to every one of the seven 
listed research or research-related functions and, because it has been 
identified as being*' in short supply, it should perhaps be isolated as a 
skin that is general izable to any function and therefore should be a 
basic part of any training program. 

Diffusion skills identified by the interviewees as being important 
may be placed into two categories, communication .skills and evaluation 
skilTs. It appears from the data in tables 2J3-2.15 that both^ypes of 
skills may be in short supply. Again,, these data provide information 
needed by those interested in developing a training program for diffusion 
perSbnneK It may be advisable^ei ther to send trainees JLo comnpni cation ^ 
schools (e.g., journalism, media, public relations) and also train them in 
evaluation techniques, or to accept comnunications gradates fntq diffusion 
training programs which emphasize evaluation. 

The data in Tables 2.16-2.18 indicate that certain context evalua- 
tion skills on the list may be more important than others, but almost^,all 
of the skills are in short supply. The important skills are both logical 
and en^irical in nature. Methodological skills also appear to, be quite 
important. It is evident, in additi can that skills such as designing a 
monitoring system require a great deal of training in fundamental tasks 
and also much practical experience. 

1 



?5 



■Almost all of the input analysis skills were cons^'dered to be 
important by the interviewees. The high percents of responses for each* 
skill may be a function of the small, number of skills, but., as before, 
it remains that few interviewees identified the skills on the list as * 
unimportant. The important ski.Hs which were identified as being in ' 

short supply are primarily logical skills and may b€rt result from prac- ^) 

' • _- 

tical experience, supplemented by short-term training. 

All of the skills listed for the process evaluation function were 
identified quite frequently asr being important.; TKe high percent of 
total responses for each skill was probably a function of the number df 
skills which were cons^iderpd, but, again, few interviewees labeled these 
.^^ills as being unimportant. All of the skijls were identified as being 
in short supply by the interviewees^ vindicating that there is a need for 
programs designed, to produce program monitoring skills. 

Several outcome e\^aluation skills were labeled as being very impor-. 
tant. Most of these included both logical and empirical inquiry skills. 

It is important to consider the supplemental comments made by inter- 
viewees (reported in Appendix D) . The perceptions on how persons. might 
be trained' in skills that^are in short supply were less helpful than had - 
been hoped, but ^till may be very valuable to the directors of training 
programs in stimulating creative training ideas and in allocating resources 
available for training. | 

It is alsoclear from the suggestions for additional skills that 
the lists provided in Appendix E were not complete by any means when -they 
were given to the interviewees^ Many of the additional skills suggested 
by the interviewees were thought to be important in training research and 
research-related personnel. However, further consideration must bje given 
before the original lists are revised. 



/ It 



seems desirable to have practitioners in research and research- 




related activities project training needs . for^ the future, rather than 
depending on opinions of, those removed from the practical reo-lit^iles. 
Since the interviewees are not oijly supervisors or employers df resear\^h 
or research-related personnel, but also^in most casles are themselves 

practitioners in these areas, the skills given in Appelidixt contribute 

*• * 

Infdnnation which should be^ considered when training programs, are being 
^designed to anticipate future needs; v_ 

Two additional observations shpuld be 'recorded here. First, several* 
interviewees expressed the opinion tnat evaluation was- oVerrepresented 
among the functions* SeveVaf pers^ons suggested re-combining the four; 
.^valuation functions into one;^ot;hers suggested that "context, input, 
and process" might better be s/jbsumed under a jieW rubric, reserving the 
term ''evalQation" to refer to outcome evaluation. ^Second, it was noted 
that tite list of research skills was biased in favor of be^favioral, empiri- 
cal research skills and failed to include skills relating to research on 
research methodology ger^se^and philosophicaj and historical inquiry. 
Suggestions and comments such as these were considered, along with all 
data reported herein, jn .a re-examination of skills necessary for research 
and research-related activities. The results of that re-examination are 
presented in thfe following section. ^ 

One decision that was made on the basis^ of these suggestions was 
to reduce the seven functions to four by using a single rubric— evaluation— 
to describe functions previously spread across faur categories. Thjs 
wisdom of this decrision will be re-assessed in further Task Force activities. 
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lEssential Knowledge and Skills for EducationaT 
Research and Evaluation ^ 
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Originally, it was proposed that reahity-testing of the<^Task Force 
lists^of research and research-r.el ated skills 'be followed by a discusjsion 
of essential knowledge and competencies not oVily in educational research 
and evaluation, but also in development and diffusion. ,iTi attempting 
to'cpmplete that task, however, the staff found itself to be signifi/an^ly 



./less^ sure of the "stuff" of whicK development and'' diffusion are^de \ 

' / /^^^^ * ^ 

than of research and evaluation and fir^ally reached the--^nclusion xhat | 

\ 

the specification of knowledge and skills in d^vi^topment and diffusion \ 
shoiJld be left to other authors. who fe^^^^re assured abTout the content <^ 
of these areas. ^ ' ^ " . ' \ i 



This section builds in an essential manner upon two technical 

13 ' - 
papers in the Task^Force series^. Technical Paper-I^of 1 and Technical 

14 

Paper No. 4. . In the latter paper, educational research- and evaluation 
are defined and examined in detail. The reader must seek \x\ that paper 
the* justification for many of the pcrsitions: taken here. In Technical 
Paper No. 1 , lists of skills required for educational research and eval'ua- 
tibn were presented. Much of the thinking reflected in that earlier pa^er 



1 * 

, Technical Paper No> 1 is essentially identical with the firsju-*^ 
=part of this chapter, which contains the original lists of research. and 
research -rel ated ski lis. 

14 

/ Technical Paper No. 4 is presented, in its entirety, as, Appendix J 
to this report. Tt was prepared under U. S. Office of Education, grant / 
6EG-0-70-4977 for the Conference on Teaching Research of the Oregon System 
of Higher Education. . , 
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IS incorporated in this presentation. Howel/er., inadequacijes in those ' 
earlier liif^ts hav^led to some modifications and altered approach presented 
* here. 

Specific criticisms of the earlier lists" of skills are of four 
main types': (a) the skills vary widely,, withirr >ists, in their level of 
generality -.- one skill might welTsubsume several others in the list, ' 
(b) the skills vary greatly in importance — some are^critical and orhers 
almost frivial, (c) many imfiortant^skjlls are omitted, and td) in re's^iarch, 
' the skills reflect only those skills ijmportant in the conduct of empirical, 
*behavioral' research skills necessary in other types of research are 
almost completely neglected. 

Some of the criticisms point to weaknesses that ma^ be a result of 
• the mannfer in which the lists were constructed. 'No empirical testing- was 
done to identify important skills — they were generated on a priori 
grounds through logical analysis. It can be argued that detailed lists 
< of /cills might better be developed through task analysis procedures. 

Force is currently developing such procedures in aji attempt to 
*• identify other skills not included in prior lists. In the interim, it" was 
decided to modify the earlier Tists In ways L'lat respond, at least partially, 
to the criticisms that have been made. What appears here is not represen- 
ted as a final revision of the earlier lists;^ rather, it is an interim 
, step. Some skills that seem trivial have been dropped from earlier ' ♦ 
'lists; anci some that, on intuitive grounds, seem important have been added, 
i Skills relevant to research on meth'odology have been added and more specific 

skills in- the earlier lists have been subsumed under more general ones. 
- Some rationale for inclusion of each skill is provided/ In essence, a 
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catalog of inquiry skills has been attempted that is neither. so general 
as to be useless* nor so detailed as to be mind-boggling. The result of 
steering this middle course is a list more general than tliose proposed 
in Technical Paper No. 1_, and one which will surely appear to many to be 
too general . In one opinion, however, a more detailed list based only 
on logical grounds stands little chance of being taken seriously and 
stimulating dtsYussion among a wide audience of curriculum planners ia 
educational resi(earch and evaluation, /ys further Task Force efforts at 
task analysis are accomplished, the results might well be more detailed, 
empirically derived li^ts that will provide more useful input to persons 
constructing training methods or planning specific training activities for 
educatiof^al researchers and evaluators. 

The remainder of this section is divicfed into three major parts. 
In part^ one, some of the skills necessary for the successful pursuit of 
research are listed. In part two, skills are listed for evaluation. In 
part three, essential knowledge about those inquiry methodologies which cut 
across educational research and evaluation are listed and briefly discussed. 

\ 

It should be noted that the ideas presented in the remainder of this sec- 

tion are proposed here merely as a tentative position statement included in 

\ . 15 

the interes'j of obtaining reactions from the reseain^h community. 

Skills Necessary for Educational Research , 
In considering skills necessary for pursuing educational research, 
it is important to distinguish at least three types of educational inquiry. 



This position statement was prepared by Gene V GL*'>s and Blaine R. 
Worthen and should not be construed ^as necessarily representing the opinions 
of the entire Task Force'. - • 
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The first type is characterized as empirical and Largely behavioral- Within 
this domain lie those types of educational inquiry pursued from the per- 
spectives of psychology, sociology, political science and other behavioral 
social sciences^ The second category of educational research is empirical, 
nohbehavioral inquiry such activities as fall under the philosophy of 

, education and history of education.^ While such endeavors deal most * ^ 
certainly with human behavior, they are classified as nonbehavioral since 
they attempt to go far beyond the construction of rational systems from 
the mere observatio^of overt behaviors. The third type Of inquiry is 
methodological research research on methodology itself including such 

^"well-known sub-areas as re-search on tests arid measurement, p^sychometrics , 
stdjistical methods, and experimental design. The importance\of any (par- 
ticular inquiry skill -interacts in important ways with these tkree tvpes of - 

' \ 
inqufry. What may be a critical skill in historical research may be 

incidental to the pursuit of successful formal methodological research and 
vice versa. In the^ollowing list of research skills, the importance of that 
skill for each of these types of educational research is disct^ssed; 

1. Drawing implications, from results of prior (Studies . Since 
research in any area is cumulative and evolutioiary, this skill is impor- 
tant in all three areas of educational inquiry. No reseaj^ch endeavor 
stands alon^, and no researcher can afford to ignore or slight the activity 
of carefully studying the^li terature of his field and drawing the appro- 
priate inferences, for the future course of his own work arnl^the discipline. 
Whether he be an historian, a philosopher, a psychologist, a sociologist 
or a statistician, he must be able to interpret, evaluate and synthesize a 
Jiteratur^ relevant to his area of concern. He mCjst^be abVe to identify 
^significant problems that are posed by the tradition of inquiry in his field 
and the^ccumulated^w^ks of his predecessors and contemporaries. 
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2. Formulating hypotheses or empirical question s to be answered 
by the study . Such skill is the hallmark of rigorous inquiry in any 
empirical field whether behavioral or nonbehavioraJ , Hence, this skill 

^ is important for both classes of educational research. However, foirmal 

research on techniques and methodology se^ldom involves the formulation 
- of hypotheses or the investigation of empiri.cal questions. Hence, this 
activity is' nearly absent durihg the working day of methodological educa- 
tional researchers. It therefore is an unimportant skill for such 
researchers. 

/ 

3. Procuring and managing resourced (material and human ) neces- 
sary to reach the research objective .^ The importance of such a skill 
obviously depends on the size of the undertaking. By merit of their 

greater si;^e, empirical behavioral and nonbehavioral research activities 
I 

(for example, sociological, psychologij^al , historical forms) more often 

x/ ' / 

involve the procuring and fnanaging of a significant number of human and 

material resources than methodological ^educational research* Sjuch skill 

could be so important in some areas of'empirical educational research 

e.g., sociological swvey research that explicit trair^ing in project 

- management techniques would be necessary. Seldom, however, is an inquiry 

, into psychometrics or research methodology of such size as to necessitate 

the management of any "substantial body of resources.' Thus, training such 

researchers in research management is probably not time well\ spent. 

4. Specifying data or evidence necessary for a rigorous test of 

/ / 

. the hypothesis > This skill is quite important in the pursuit ofNany 
^^^^^^.^^^ejB^irical research effort, whether behavioral or nonbehavioral. As regards 
methodological educational research, the necessity to be so skilled is 
unimportant. 
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5. Identifying the population to which results should be general- 
ized, and selecting a sample of the population . Skill in identifying 
relevant populations and sampling them representatively is crucial to 

many forms of empirical behavioral research* Sociological survey research 

r 

is almost totally dependent for its utility upon this step having been 
successfully completed. The historian, however, seldom has control over 
those events which provide evidence for his conclusions. The educational 
historian of the past and, to a lesser extent, the contemporary historian 
arp dependent upon those traces of the past events which were fortuitously 
left behind (in diaries, private, correspondence, e^c.) by a handful of 
extraordinary and nonrepresentati ve individuals. Although the histt)rian 
!^ is greatly concerned with the evidence which survives the rigors of time 
and comes into his hands, he seldom is forced to evaluate the generality 
- of^uch evidence in the same manner as the sociologist. Whereas we can 
identify a very limited number of skills from statistical methods for the 
evaluation of the generality of sociological evidence, within histpry the 
generality of evidence must be evaluated^ar less formally and often on 
an ad_ hoc basis Skill ^5 is incident.^1 to the methodological resear- 
Cher, except insofar-as-he-eheoses to study the'process of general izability 
as a formal procedure. 
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^^See Beach, M,,A History of Education, Review of Educational 
Research , 1969, 39, 561-76 (especially 565-69), 



6. Applying an experimental design and recognizing and cont rolling 
threats to its validity . Most empirical behavioral research conducted 

in the laboratory depends for its success upon this skill being success- 
fully exercised. The words "applying an experimental design" connote 
the manipulation of variables as in traditional experimental psychological 
research. .In no real sense is an experimental design applied in nonbeha- 
vioral research into the history or the philosophy of education. Hence, 
this skill has no relevance for such endeavors. In methodological educa- 
tional research, the application of experimental designs and the recogni- 
tiqn and control of threats to the validity of experimental inferences is 
not a requisite skill. However, for some small numbe^r of such formpl 
researchers this skill becomes the object of their direct inquiry.' 

7. Identifying the classes of variables for, measurement . The 
identification of behaviors for measurement is crucial in psychological 
and sociological research on education. It is far less crucial for his- 
torical and philosophical research, and as ^^a requisi-te for valid inquiry 
i^ is totally nonexistent for methodological research. 

8. Selecting or developing techniques of measurement . Thts 
skill IS a cornerstone of mych empirical behavioral research. Coupled 
with the next skill (#9), it constitutes the most critical stage jin the^ 
pursuit of empirical behavioral inquiry. The latitude existing fir the 
selection of events in an inquiry is greatly restricted when one moves 
into the areas of history and philosophy; there one must more often be 
content with observations which one has not chosen on a basis for pursuit 
of the^study. Since formal educational research as methods and techniques 
is generally not empirical, this skill is not essential for such types 

of inquiry. 
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9. Assessing the vali dity of meas ureinent techniques > Scarcely 
any skill could be more important any empirical inquiry than this one* 
The validity of the findings depends in large part on the researcher's 
skill in embodying the general Xv)nstructs of his inquiry in a set of 
rneasurement techniques, 

10, Utilizing appropriate data-gathering inethods (tests, inter - 
views, analysis of documents, etc) . This skill originally appeared as 
V-i: "Using a variety of data-gathering Instruments.'' If stress is laid on 
the word ''variety" in this statement, the skill probably receives a rating 
of unimportant for all three areas of educational inquiry. Most educa- 
tional research ought to be narrowly focussed; one problem that has 
impeded' educational research is that many researchers have a "bag full 
of methods" and use none of them well. Although all types of empirical 
researchers must' be skilled in the use of some data-gathering techniques, 
-Only, rarely will a research9r*s inquiry range so broadly that a wide 
variety of data collection methods must be mastered. (Some large and 
complex areas of sociological research could prfeperly be exempted from such 
'a statement,) However, this is pot to argue that researchers should master 
only a single method and apply it willy-nilly in all contexts; such behavior 
would fall under the "law of the instrument" fallacy deplored by Kaplan 
(1964). Researchers must learn well a small number of data-gathering 
methods appropriate to the problems in their disciplinary base. This 
skin is important for both types of empirical inquiry. 
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1 1 • Understanding the general ro1e> types > and assumptions of 
statistical techniques and drawingj^n such knowledge in using appropriate 
techniques of data analysis . This very general skill is relatively unim- 
portant for empirical nonbehavioral researchers, among -whom we class 
philosophers and historians. It becomes important for empirical beha- 
vioral researchers such as sociologists and psychologists. It is a critical 
skilT for methodological researchers who are frequently called upon to 
advise empirical researchers of many persuasions on the proper analysis 
of research data, 

12, Interpreting and drawing appropriate conclusions from data 
analysis . Such a sji^ll is the heart of good empirical behavioral research.' 
Since data analyses are substantially less formal in philosophical and 
historical inquiry this skill is only of average importance for such 
researchers. This skill is given a rating of unimportant for researched 
on methodology and techniques; ito's assumed that ^they are exempted from- 
the duty to interpret data for ineir clients in terms of its substantive 
importance within the client's field. 

13, Reporting research findings and implications . This final skill 
is of primary importance for aU types of edutational ^r^esearcher. Science 
is necessarily public an3 hen^e the act of publication is important in its 
growth. By exercising this final skill, the researcher brings, the inquiry 
process full cycle,. His contribution thus enters the li terature* to be^ 
drawn on by his colleagues in the pursuit of new knowledge. ^ ' 

Skills Necessary for Educational Evaluation ) 

In th^'s section ten general skill areas are identified as important 
to the successful pursuit of educational evaluation. Some may feel that ^ 
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the list excludes important evaluation skills of the type earlier discussed 
under such categories as "situation analysis, input evaluation, etCi" 
The unease about whether these are appropriately thought of as evaluation 
skills per se or as other important (but nonevaluati ve) inquiry skills is 
reflected in Technical Paper No, 4 (Appendix F) . Only those skills which, 
at present, are felt certain to belong to educational evaluation are 
included in the list. Decisions about the appropriateness of including 
other "evaluation" skills from the earlier lists must await further - 
elaboration of Interrelationships among Inquiry activities. 

1 . Budgeting and managing human and material resources . Many 
educational evaluation projects are large, multlfaceted endeavors^ entail- 
ing a financial budget exceeding that of most educational research projects. 
In such projects, material and human resource's must be efficiently managed. 
It seems necessary that those who would engage in educational evaluation 
at an administrative level should be trained in some of the techniques of 
project management and financing. 

2 Identifying at -appropriate levels. of generality the goals of 
the -program to be evaluated . It cannot be assumed that .th^ goals of a 
program which one wishes to evaluate are known or stated in advance of 
the evaluation. An important activity in getting an evaluation underway 
will typically be the attempt to elicit from the responsible persons the 
goals and objectives toward which the program is directel^. The identifi- 
cation of these goafls is more than a routine activity of solici ting verbal 
statements o^ goals from program personnell Done properly, it can easily 
entail some of the most sophisticated technology of survey^search and 
Interviewing. What many evaluators experience as frustration in ttieir 
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attei^jts to elicit statements of goals from program personnel is actually 
evidence of their own expertise in such activities. 

3. Assessing the value of program goals . This activity is the 
earliest explicitly evalua^tive act of an evaluation. In an evaluation, 
the goals of the program mu^t not be accepted at face value, but must be 
regarded as elements of the program requiring direct evaluation, much as 
one evaluates program operations and outcomes. In some instances, the 
justification of goals must come from empirical research in education or 
in social sciences. As an example, a program aiming at the inculcation 
of reading re^idiness skills in five-year-old children immediately raises 
the evaluative question of the facilitative effect on reading of the 
attainment of. reading readiness. The evaluator must know how to search 
for a justification of these goals in empirical research on reading instruc 
tion. He may find, .for example, that the case for reading readiness has 
never been adeq,uately established through empirical research. Hence, he 
may legitimately raise a question of the justification of the program 
objective. In other instances* one must turn to nonempirlcal nonbehavioral 

N 

disciplines such as philosophy and law in seeking to evaluate program 
goals. A s<^hool system may set out as an obj.ective to teach all children 
in the'school the Christian ethic. The evaluator must be sensitive to 
and should" raise the issue ^ either pr>vately with program personnel or 
publicly in his report, of the legal and* phildsophfcal problems concerned 
•with the separation of church and state in the United States. Assessing 
the value of program goals is an activity likely to carry the evaluator, 
far beyond typical concerns with behavioral statements of objectives, 
criterion-reference-d tests, and statistical analysiVr" It^require^' that 



he be educated broadly in the social sciences and philosophy, and that 
he be responsive to questions of value which are broader than those he 
can ever hope to Investigate within the span of one evaluative study. 

4* Translating broad objectives into specific^ observable 
objectives * General goal statements must be operationalized into specific 
statements of objectives. The onus of making this translation lies clearly 
with the evaluator who possesses the technical skill for doing so and not 
on program personnel to whom the language of operational ization and 
behaviorism is foreign and unfamiliar. Of course, the translated objec- 
tives must be redefined by-^program personnel to* prevent unconscious bias'^s' 
of the evaluator from producing operational objectives different in intent 
from the broad objectives with which he began* 

5* Identifying standards or norms for judging worth . The 
measurements and observations taken in an evaluation cannot be translated 
..into judgments of worth without 'standards or norms. The formality o? 
these standards and norms may 'vary greatly, but nonetheless a standard is 
implied whenever a judgment of worth is derived from an observation. The 
evaluator must be sensitive to tfie various standai^ds which different groups 
us^e in judging worth. From among these standards, he mu^t choose those 
which can best be justified. Standards may be either internal or external 

a particular evaluation. Extern..! standards ar? represented by collateral' 
data with which observations and measurements are compartru in deriving 
evaluations and judgments of worth. For example: a school system may 
desire racial balance in its schools and may have attempted, tg achieve . 
this bal ante. through various means. The observation that 75 percent of 
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all Negro pupils attend schools where less than 25 percent of the student 
body is Negro is an observation not yielding immediate eva'luative meaning 
since no standard for judging this degree gf racial mixing exists. A 
standard for judgment could be found external to .the evaluation in data 
in Equality of Educational Opportunity . There^it might be found that in 
a representative sample of school districts across the natio^n, 95 percent . 
of Negro pupils attend schools which are 90 percent Negro in population. 
By bringing these external data to bear on the observations from the 
evaluation, it is clearly seen that a much more satisfactory racial mix / 
was achieved within the school district in qu^tion. The wKole-area of 
comparative experimental design is a means cvr establishing internal stan- 
dards by which the worth of qertain activities can.be judged. A program 
is pitted against an alternative program /and the worth of the former 
program Is measured vis- a- vis the outcomes of,. the l^atter. 

6. Monitoring the prOgcam to detect jdeviations from design or 
specified procedures > , It 'is, of course, important to know what one 
evaluates. It is insufficient to accept mere' labels when one has invested 
large portions, of time and money in the observation and judgment of outcomes. 
It is necessary that a program-be monitored througt]^site visitations, inter- ^ 
view techniques, survey research methods, etc, so that the evaluator is 
clearly aware of the degree to which the program proposed was made opera- ' 
tional. It is misleading to pronounce a judgment of "unworthy" on a team 
teaching program if team teaching was never genuinely -attempted. 

7, Selecting (or developing) and using valid techniques of 
fteasurement to yield. information on outcomes . The worth of an educational 
program lies In its-outcomes. It is crucial that the proper outcomes be 



validly measured. Objective, yalid data on program performarjce is the 
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si ne qua non of any ^justifiable evaluation. The. evaluator tnust have skill 
in selecting those techniques that will reveal objective data on outcomes 
where objective data are possible. He" must know when a measurement tech- 
nique threatens to misrepresent a set of behaviors. 



8. Employing appropriate techniques of data analysis > The 
evaluatorvmust be broadly knowledgeable in the are^ of statistical data 
analysis. He must have al clear understanding of the' fundamentals of a 
variety of data analytie^echniques , He must know when a factor analysis 
bears critically jon an evaluative question and when it i§ mere window 
dressing for a flashy but Superficial evaluation. 

9, Making recommendations as a result of tf<e evaluation . The - 
evaluator's respons-ibility to evaluart^ ^does not end with the collection, 
analysis, and reporting of data* The data do not speak for themselves/ 
Surely the perceptive evaluator acquires a valuable perspective on the 
educational program being evaluated through long and intimate assotiat.ion 
with it and by merit of the special pet'spective he brings to the program. 
To fulfill his total eyaluativ^ responsibility, he must make the subtle, 
and personal inferential leaps Ifrom those data he has gathered and those 



results he.hhs observed to his bersohal recommendations for the |Conduct 
>or the continuance of a program.1 This is not a skill easily taught. The' 
activity draws upon the^ accumulated experience, wisdom and judgment of 
the evaluator, 1 . T ^ 

* 10*. ' Writing the evaluation report . Drafting the report of an 
evaluation for the relevant audie'nce or audientes is an activity quite . . 
unlike writing the report of a research project. The format for the report 
of research tends to be stereotyped within a discipline; publication ^ 
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manuals clearTy spe>l but the pafrts of a research report and the conven- 
tions one ought to employ. Detail and completeness are valued frlghly: 
Subjective opinion Is downgraded in a research report. The researcher 
Is communicating with? his colteagues who already have consklerable 
experience and background In the area being discus,sed;. The'^wrltlng^of 
the evaluation report la a different matteV altp^gether. The evaluator Is 
typically communicating to anUudience whIcK -does hot share his perspec- 

o r I " - f ^ ^ 

tlve, hISL grasp of technical topics, nor;;fi1s interest In technical details. 
The responsibility to communicate fin^fngs rests more'heavily .with the * 
evaluator than with the research^v The eval Uator will have to adopt a ^ 
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nontechnical language. He mus^ refrain from over-reliance on tabular 
presentation of data analy^^s. He must avoid discursive commentary on 
test validity and refiab^ity and other topics which his audience will ^not 
find central to their/concerns. At' this, final stage of the eyalyatjLon^ 
endeavor, the eval.u^or will play a role much ijiore akin to tj)^ journalist 
than the scientist. - •* 

* ' ' ' 

Essential Knowledge about Educational 

Research and Evaluation Methodology . • - / " 

For the ^Ik pf educational researchers and evaluators; methodolo- 
gical knowledge need consist of little more ^han the ability to ppduce 

5> V ' , ' 

thretf pages of coherent exposition on e^h of approximately 50 topics. 

17. 

It is an insidious form of -the '''academic fallacy" to th-ink that all ^ , 



^''scrivfen, Micha^-l. Education for survival) Ch. 3 in, The Ideefl 
School , Gloria Kinney (e..). Wilmette,' Illinois: .Kagg Press, 1969. 



educational researchers should have a specialist's grasp of research , . 
techniques, acquired through the studyof method in its own right. The 
methodlogical knowledge*" required to hold officers a practicing educational 
researcher or ^valuator is far less' than 'typically imagined or admitted 
to by me'thodOlogists. . Knowledge of the effect of platykurtosis on the 
distributi^on df eSytimated vari ancVcwonents is no moreyecessary to the 
instructional researcher than is knowle^^of the effect of exposure to . 
nonexemplars on acquisition of djsjuncV've concepts^necessary for the 
mpthodolo^ist. To act otherwise hobbles methodologtcal research and 
retards instructional research." In truth, educational researchers need to- 
know only a little' methodology relevant to the disciplinary base from 
which theyjaunch their investigations: evaluators may need to know a 
little about mor^ types of methodology, since they are forced 'to work more, 
across disciplinary bases, but even here depth o^ knowledge about particlila 
methods is not requisite. If this ^sounds like an argument for a super- 
ficial knowledge of !a body of methods and techniques then the message has 
^come through .clearJy. Irrational i^jsistence in the mastery of methodolo- 
gical esotetfica only breeds guilt and avoidance* reactions in educatioihal 
researchers aVid evaluators. The ability to write a few pageis'in explana- 
.tion of each oV the following concepts represents the^essential core of 
knowledge of edCicational research and evaluation methodology. 

Statistics 

1. Library knowledge: names of major books, their authors,- 
and some familiarity with' content. 

2. Descriptive techniques, their definition and interpretation. 

■ including measures of central tendency, variation, correlation 
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and prediQt'ion (including multiple regression and partial 
correlation) . 

3. Contingency table analysis of categorical data. 
4t. Major schools of thought on statistical inference (Neyman- 
Pearsonian, Fisheria^, Bayesian, likelihood estimation), 

including principal concepts thereof. , 

5. Nature and use of the general linear motjel including least- 
squares estimation and distributional theory, which includes 
analysis of variance methods. ^ i 

6. Fundamental theorems of Jinjte sampfe space probability theory 
• (addition rule, multiplication rule, etc.). 

7. Definitions and\roperties of the principal continuous (normal, 
chi-square, t, F) and discrete (bi^ipmial, multinomial) pro- 

o 

bability distributions. 

8. Permutation theory and Mon^e Carlo methods. 

■ ■ ' . . r 

9. Nature and purposes of the following variations on simple 
random sampling in suryey research: stratified sampling, 
cluster sampling, multi-stage sampling. 

10. Consequences of failure to meet assumptions of principal para- 
metric inferential techniques. 

/ 

Experimental Design * \' • 

'l. Library knowledge: names of major books, their authors, and 
some familiarity with cont^ts, 
2* Randomization as^a means of experimental control and its 
relationship to inferential statistical methods. 



3. Factors Effecting the internal and external validity of . 

^ ■ <• 

experimental and quasi -experi,mental designs. 

. ■ .4. ° Definitions o'f fixed-effects^andom-effects and mixed- 
effects^ designs; crossed and nested factors. * * 

' ■ \ 

5. Covarying, blocking and stratifying as means of increasing 
» . — . 

precisibn of estimation in experimental designs. . 

6. Purposes underlying the use of randomized blocks, Latin 

' ' ■ i ... 

square, Greco-Latin square, fracti'onal factoi^ial, incomplete 
block, etc., designs. , - ' 

7. Nature and^ problems ,in the use and analysis of "repeated 
measures" designs. . • ' 

8: Effek of measurement erroft)n the precision (power) of 
- > aa experiment. * .^^^^^^^^^^^ 

' ) * ' 

Psychometrics , . . *^ * 

< 

1. Library'knowledge: names of major, books, th^eir authors, and 

some familiarity with contents. 
• 2. - -Fundamental postulates of classical tnie-score theory (bo^h 

the "theory of errors" and the "the/ry.pf pa'rallel measures.")- , 
3. Fundamental theorems of classical true-score theory (reliability, 

coefficient, variance error of measurement, correction for ' 

2 2 2' 

attenuation, relationship of test len|th to Oj^. o^, and 
Pxx^* 

- 4. Types of test reliability and validity. 
5. Reliability of sums and difference scores. 
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6. Measurement of change or "gain" (definition of "gain," relia- 
bility and validity). 

♦ 

7. Fundamental postulate and theorems of common-factor analysis/ 

8. ' Four "factor analysis" models: components analysis, image 

analysis, canonical factor analysi\, alpha factor analysis/ 

9. ^ Methods of factor rotation (orthogonal) and transformation 

(oblique): varimax, equamax, quartimax; oblimax, promax, 

/ 



Harris -Kaiser. ' >^ -"^ 



Measurement 



V. Library knowledge;' names of major books, their authors land 
some familiarity with contents. 

2. ^ Major forms of assessment of knowledge and cognitive skills 

• including multiple-choice, completion, free-response, ranking, 
matching, etc., formatsr; 

3. . Primary fliethods- of assessing attitudes, including Likert and 

Thurstone scales, interests and social perception, including 
semantic differential and Q-sort. 

4. Fundamental theorems on the differential weighting of test 
items (particularly Wi Iks' s theorem). 

5. Properties of the major test-score scales including T-scores, 
z-scores, CEEB scores, ration and deviation IQ scores, grade- 
equivalent, scores, and percentile scores. 

6. Definitions and properties of nominal , ordinal , interval and 
ratio measurement scales.- 
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7. The nature and implications for< reliability and validity of 
response sets* 

a. Construction and use of rating scales, inclucfPng methods of 
" assessing, rater agreement.. 

General Considerations in Research and Evaluation 

1. The distinction between different -types of inquiry — e.g., " , , 
basic and applied (conclusion-ofiented vs. decision-orieked) 

• - research, formative and sunmativ^ evaluation. . ' 

2. ' The nature of theories, models =and paradigms in the social .; 

sciences. 

3. The nature,-&f^th'e phenomena studied by psychologists, soci6-. 
logists, economists, cultural anthropologists, political scien- 
tists, and philosophers (for example, externally reinforced 
individual behavior (psychology), social organizations 
(sociology), exchange of valued comnodities (economic?), 
cultural norms (anthropology), authority relationships (politi- 
cal science)» and linguistic meaning (philosophy). 

Conclusion 

The listing of skills and knowledge essential to practicing re^searc)!- 
ers and evaluators in education has led us to five major conclusions. 

1. If one accepts this portrayal of skills and knowledge^ essen- 
tial in research and evaluation as even partially correct, it is clear that 
there is too much to communicate to trainees in the time normally spent 
in a graduate program (e.g.. consider the indepth knowledge of -a discipline^ 
essential for a researcher). This mandates more attention to recruitment, 
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since it i^s obvious that selecting trainees who already' have some of the 
requisite knowledge or skills woald largely eliminate this problem. This 
leads logically to the conclusion that universal recruiting from other 
disciplines is highly desirable. Even the selection of students with ? 

# * V . « 

relevant research and^ evaluation (not teaching) experience or a.^good 
IJberal Jrts, ed^ation would be helpful. 

2. Current graduate programs do not effectively train students 
in many of t^he skills and much of the knowledge listed here as essential. 
A'tradition exists, primarily in universities, of teaching statistics,, . 
measurement, ejcperimental design,_and certain other areas listed earlier 
in this paper. However, many of the essential skills and knowledge are 
not being well -taught in current training programs and, as a result, many 
current. occupants (k^. .evaluation and research rplfs are poorly trained for 

those rOle^.* * ^ , ' > _ 

' 3. 7rainers need to either (a) upgrade graduate training programs 

♦ a 

to Where they will focus on at least the minimum essentials listed herein, 
or (b) develop more effective Alternatives to graduate programs as part of 
a new long-range strategy for improving -the training of educational 

t 

researchers and evaluators . 

- 4. In the interim period,- trainers must depend on ancillary 
training strategies to teach mafny of the essentials. Specifically, one 
might look to short-term training and retraining strategies such as work- 
shops, institutes, and self-contained, exportable, progranmed materials. 
All, of these strategies have promise of reaching broad audiences and pro- 
viding training in^ skills and knowledge now going begging. 



5. Many of the skills listed herein might be best learned throiqh 

T8 

apprenticeship training- in educational research and evaluation.. Por 
example, "drawing approprl-ate conclusions from data analysis," "assessing 
the value of program goals," and "writing the evaluation report" are * 
(perhaps appropriately) not included in formal training, programs. Those 
who possess such sifills generally obtained them through expWience, often 
'under the tutelage of a senior ^searcher insightful enough to make of the 
apprenticeship a genuine trJiining experience. 

In the words of Sibley (1963): 



No amount of formal instruction in methods and • 
no amount of discusston of others' research can take 
the flace of the first-hand experience of undertaking 
to translate an unstructured situation into a problem 
or problems Amenable, to" scientif^ic investigation, and 
" then proceeding to seek solutions. Every candidate for 
an advanced degree in a scientific discipline ought to 
serve an apprenticeship in research, beginning as soon 
as he^has completed a necessary modi cum of formal 
study of methods. The term aporer^iceship is used 
here in default of a better one to denote learning 
by working under the personal direction of a mature 
professional person. (Sibley, 1963. p. 37) 



^^See Technical Paper No. 20 in the AERA series, in this regard, 
as well as Appendi^c F. . 
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ANALYSES OF AERA EMPLOYMENT SERVICE DATA AND 
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ANALYSES 0<^ AERA^ EMPLOYMENT SERVICE DATA. AND 
' IMPLICATIONS FOR RESEARCH TRAINING 



A major objective of the 1969-70 Task Force project was to provide 

information ofi supply and demand of educational researchers to those who . 

"Rave responsibility for developing research training programs. Of parti- ^ 

cular concern were data oh areas of special izatj on in research and research- 

related activities in which there are sHortages of trained personnel- The 

telephone interviews discussed in the previous chapter were used co collect 

^data on such shortages, .as perceived by employers and supervisors of 

^» ' # 
research personnel- It was considered qssential , however, that some infor- 

mation be provided on the competencies, required by; employers for actual 



. positions open and competencies possessed by^ppiicants for those oositions. 
To that end, the Task Force staff examined existing data f»tom the 1968, 1969 
and 1970 AERA employment service operated concurrently each year with the 
Association's an^jjal meeting. rj? 

The results of that examination are discussed in this chapter. 
In the first.v three sections, analyses are presented, by year, of supply - 
and demand daia'in particular areas of competence. The fourth section is 
an examination of geographic dat? on vapancies and applicants for~a"ll. 
three years..' The last sectibn of the chapter is an attempt to analyze 
trends in the data across the three years studied anAj^o relate the <• 
employment service data to^tNe information onji^eSed skills Whicf\. resulted 
from Ishe telephone interviews'. — 



ERIC 



• 102 



_ • 

An Analysis of 1968 AERA Employment Service Data ' 

Eroplovment Service Forms ' " • 

The "profe^ional order form" usecf'by empl oyers^and the "professional 

application form" used by applicants in the 1968 AERA placement service v 

are shown in Appendix 6. Both are standard convention empT»yti«nt service- 

\^ ' . forms-^supplied by the U. S. Department of Labor. A. variety of data was \ 

contained on the coippjetsd forms; the most pertinent was data on area{s) 
" ^ — . ^ 2 

of competence, recorded on both forms. Specifically, employers were 

' required to listttji^ a*rea(s) of competence necessaryToFeacfi available 

position and'^ach applicant was required to list his area(s) at competence. 

No specific description 6f the type of response desired was provided to 

either employees or appl i ants; consequently, there was considerable varia- 

tion in the lei/el of specificity in the responses. Most Of the responses 

were quite gldbal (e.g., statistics). Data. about applicant competencies 

w^re self-r«>nort ^ta and, therefore, sObject to the difficulties inherent 

ift-such data.^- The$0^ limitations notwithstanding^, it was lel^that^ coclW' 

parisons* of applicant and employer-data would. provide useful infqrmation 

to tfte Task Force. • ' 'J. - . 



' Vhis placement service operated during February 8-10, 1968 in ^ 
Chicagx), v ., 

^O^ier data on "geographic distributton of available positions and 
geographifcWeferenceSjOf applicants are reported later In this chapter*. 

^An a^Ssment of the reliability of tt»e self-»^port data -from' 
the 1970 AERA employment service is contained in Appendix J,; an assessment 
which also has implications for the^968 and 1969 daU.'. 
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Procedures • , " v 

The procetfures for. organizing and analyzing the data are repor- 

* *' 

ted below, ^ ' ^ ^ 

y^,^ Each applicant fomi was -reviewed and each area of competence 
— • ' • J 

mentioned was listed alphabetically in the precise form in which it • 

* 

appeared on the form. Differirtg statements of areas of competence were" 

all listed separately (e.g7, "tests and measureme.nt" was listed separately* 

from "measurei#it"); identical "statements were simply tallied. ' ; 

' - ' • , , / 

~ 1 The resulting listing* of applicants' areas of competence was/ 

collapsed into t^ie logically derived categories used below to report the 



results. 



S 



"' 3. 'Each application form was also categorized as to whether the 

■applicanf 'seemed .to be seeking a research »positi on, a research faci 11 tative 

position or a nonrWearch position.* For thisv categorization, all relevant 

I ■ ' \ '* i , . 

ihformatiorTon the \f orm {e.^., areas, of comjDjetence, special Information 

lis^ted, prior pfofe^sjonal duties ahd. caree^ eyo1utiQn)^was Used to assist 

in n\aH(ig a Judgment, /f ^ • 'i <• 

' ^ 4. Each employer fomi was reviewed^^nd eac|i» area of competence 
Ifs-te'd^alphabeticany, using the same process a^s reported above for' 



I 



V "research position" was operationally ^fined «s.one in which 
there was some indication that, the applicant desired <}r was primarily m 
prepared for a position in which he ^ould participate in any way m research 
or a research-related activiUy (evaluation, development or diffusion;, 
'Vesearfch faci li tative "position" was one in which the applicant did not 
qualify for the above category but indica-ted an interest in a position in 
which he could either" teach ^or'ackninister research, evaluation, development 
or diffusion content or activities. Whfip none of these indication? were 
present, the applicant was ckegorize,d as seeking a nonresearch position 
Althf'ugh this categorization Wocess was not based ^t- clear criteria., »• 
aqreenent between "cat^gorizeVs" was checked on an eigkt percent sample 
of applicant forms and was foilnd to" hav#*86 percent perfect agceeme/it^. 



applicant foms. The resulting lists of areas of competence required 
by employers were, coll apVed into the saiiie set of categories usedjfl^ 

applicant?. . • v 

5. E^di employer. fSnn^w^s^ also categorized .as tp whether ttte 
position was a research, research facilitative or nonresearch position. 
! ' 6. Employer forms Were also •categorized into ten ^flstitutional * 

settings and cf^ss- tabulated against "areas of compet^ce."/ ' , • 
. 1 . . . 

i 

Results • s. * , ' ■ 

Employers regist^d 769 vacancies with *he 196& eifi|)loyment service; 
' \ , ' ' ' 

811 applicants registered with the service.^. Although this appears to, 

rep^sent a* reasonable- balance between supply and demand, it| is only when_ 

the vacancies and- applications are aaa^lyzed in terms of cwnbetencies and 

institutional settings .tha<>peci.ftc supply--demirid discr6paficies cah be . 

identified. Such analyse/are presented in the tables that follow^ 

Tabulation of 661 of 769 vacancie|, by institutional setting and , 

by a detenminatiori^ of whether the vacancy, was 'for a resaarch, research 

facilitative or nonresearch posiyjon, is shown in Table. 3.1. / 



' Winitions were comparable to those listed for applicants in, 
t«e previous footnote; ^however, the employer forms included more specific 
data- about the extent to whicfr researCh^^or research^related activities were 
a part of the position eithei^ through dfK^ct- involvement or ^f aci 1 1 tati on , 
through relevant 'instruct! ond1\or administrative roles.,. The Pftcpnt of 
•perfect agreement 'between the ty/o "categorizers" >/as^96 percent on employed 



forms. ^ ' 1 ' • 'i 



^he total nunfcers in the tables (fa n«(t corresponds to Jhe total^^ 



Tiumber of Vacancies or applicants registered \ith t|lf employment ?ervi^^^^ 
fbr two reasons: (a) 'some foms were blank, illegible or uninterpretable 
on critical itite, and (brsome ite.ns permitted •n"ltiPl?^»;^^ponses- for^^^ 
each applicant or employer. Each table includes a note explaining which . 
factor ojjerated to creit^ sucji discrepancies, ; , 
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Tabulation of specific areas of competence for the 661 
vacancies for which competence information was available is shown 

instable 3.2. Notfe'that in this table tnuHiple competencies were 

.;/*•■ 

Ii5;ted for most of the 661 vacancies, resulting in. a total/ num'ber ef 

/ 

.competencies recorded hgre -far in excess of the total number, of vacan- 
cies. Note also that the same 108 forms ^^xcluded from Table'3.1 were 
excluded .from Tab.le 3.2* - . ' 

SeV^^n hundred seventy-six of the &11 applicants listed areas ' 
of competence on their appli^tion forms. In Table 3.3 a comparison of 
specific "competencies ^ey listed with competencies listed for vacancies 
is presented, tabulated by relevance to research and research-related,; 
(areas. Note again that in this table multiple competencies were • 
listed by many .employers and applicants, resulting in a total number of 
competencies recordecj here that is far in ex.cess of the total appli- , 
cants or vacanciai, ^ . ^ - 

*A comparison of the number -and percentage of vacancies \ 
that are research, research facil itative and nonresearch with. the 
number of applicants whose competencies seem to prepare them for such 
positions is/shown in Table 3.4. / , 

eich applicant was also asked to indicate the type of organi- 
zation in 'which he preferred employment: academic, government or 
other. The response was disappointing: 299 ^f the '811 applicants 
left the it«n blank. Another 92 checked all options. This left' only 
420 for whom the choice among the categories, was clear-cut. They 



were distributed as shown .in Table 3.5. 
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Table 3.4 

I Number and Percentages of Vacancies and Applicants in 

1968 Employment Service: By Relevance to 
Educational Research and Research-related Activities (RDDE) 



Vacancy or Applicant Vacancies, Applicants 

Competencies are: * H % H % 



1. 


Directly related to RDDE 


223 


34.0 


469 


60.4 


2. 


Facilitative of RDDE" 


99 


15.0 \' 
■ 51.0 \ 


186 


24.0, 


3. 


Not related to RDDE 


339 


121 


15.6 




Total 


661 


100.0 




100.0 



^ Table 3.5 

' Applicant Preferences for Employment by Institutional Category 
/ Compared with Vacancies by Institutional Category 



Vacancies Applicants 
Institutional Setting N % N % 



!• Academic 


\ 

\ 


464 


70.2 


409 


97.4 


2v Government 


66 


10.0 


2 


0.5 


3. Other 


\ 


131 


19.8 


9 


2.1 


Total 




661 


100.0 


420 


100.0 



\ 



/ 



Discussion 

Before attempting to derive Implications from the data pre- 
sented above, several factors that limit their interpretability should be 

♦ 

mentioned. 

First, there is no way to know whether employers and applicants ' 
used the same referents in listing areas of competence. Thus, comparisons 
of competencies listed by employers and applicants may be distorted, to 
an unknown degree. For example, there is an apparent discrepancy between* 
27 vacancies for which "instrument construction" was listed as an area 
of conpetence and one applicant who listed that activity; that discrepancy 
may be an artifact created by employers using the rubric "instrument 
construction," while the applicant lists "psychome tries" to describe 

the same phenomenon. 

Second, lack of knowledge about who uses the employment 
service creates uncertainty as to how to interpret the data. 'Apparent 
discrepancies between supply and demand may be attributable- to differ- 
ences between enployers and applicants who use the service. For 
example, the apparent imbalance between vacancies and applicants in 
curriculum development may merely reflect the fact that. persons wish- 
ing to fill such positions typically use ASCD or some other organization, 

while applicants "hedge their bets" by registering in as many employ- 

I 

ment services as possible. 

Despite these limitations, several trends' in the data appear 
' Strong enough to warrant consideration. 

First, it is not surprising that over 70 percent of the vacan- 
cies were in'^cademic settings. The AERA annual meeting and other 
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professional association conventions are prime recruiting grounds for > 
academic administrators. It is somewhat surprising, however, to find 
that over 97 percent of the applicants who expressed unequivocal 
preferences opted for a position in academia. The large numbers of 
positions open in government agenci^, schooU, regional laboratories, 
etc., seem to have very limited appealUo applicants in the AERA 
employment service. / / 

'Second, in- view of the very generous criteria for including 
a vacancy in the research and research fac'ilitative categories, it is 
interesting that over half the vacancies "were in no way related to 
researt^h or related activities. Conversely, only 15.6 percent of the 
applicants were included in the nonresearch category. Applicants seemed 
tO/be much more research oriented than did the employers. Stated 
differently, competition for research and research facilitative posi- 
tions appears to have been keen in 196i8. 

Third, some discrepancies between the number of employees 
listing certain areas of cot^etence and the number of vacancies 
calling for those competencies are large and seem unlikely to be 
artifacts of the way the areas of competence are listed or of the 
type of user of the employment service. For example, there appear to 
" be somewhat more vacancies in research methods than there are qualified 
applicants to fill them. This is especially true where survey research 
is concerned. It seems that virtually no persons with specific skills 
in this area were prepared in 1968, although the need was patently clear. 
Perhaps persons who listed "educational research" or "research methodology 



were skilled in applying sociological survey technicjues, thus reducing 
the discrepancy, but there is no basis for such an assumption in -these 

dj|ta, ^ 

There were raore than three vacancies in evaluation for every 
applicant prepared in this area. It appears that the evaluation 
mandates of Titles I arW III of *the Elementary and Secondary Education 
A^t of 1965 and the increasing trend toward accountability may have 
had an effect in 1968 and necessitated more attention to training, in 

> f " • ^ 

this area.. 

' Although tht °dis;«;repancy is not startling, it is interesting 
to note that the need for an increased supply of developers and 
diffusers predictid by Clark and Hopkins' (1969) had not evidenced 
itself in 1968, at least among users of " the AERA employment service. 

Other areas in which far too few people were prepared in 1968 
to meet the demand included some areas of measurement, advanced 
statistical techniques (e.g., factor analysis, multivariate analysis), 
systems analysis, research management and educational sociology. 
Conversely, there was an oversupply of persons prepared (in comparison 
with relevant vacancies) in educational and schoQl psychology, v 
learning and experimental psychology, guidance and counseling; curri- 
culum development and teacfcier education. However, these latter 
discrepancies seem more tenuous and more likely to result from the 
clientele using the AERA employment service. Employers desiring to 
hire counselors, curriculum directors, and school psychologists' may well 
be more inclined to ff&is their recruitment efforts on conventions' of 
professional associations such as the Associatitjn for Supervision and 



( 



Curriculum Developmpt, the American Personnel and Guidance* Asso- 
ciation or the American Psychological Association. 

The data and discussion in this paper must be viewed as 
tentative pending the analysis and reports of comparable data for 
1969 and 1970- » 



An.Ahalysis of 1969 AERA Emgldvnfefit Service Data 



Employment Service Forms ' 

The "professional placement order form" used by employers and 
the "professional placement application fonif used by applicants in the 
1969 AEI^A-placement service'^ are shown^in Appendix H. Both aVe standard 
convention employment service forms supplied by the U. S. Department- of 
Labor: although they are different from the form? usetl during the 1968 
and 1970 employment services. A variety (Jf data was contained on the 

' 8 • ' * ~ 

completed forms. , ' « * ^ 

For the purposes of this analysis. /the shift from the 1968 form 
to the 1969 form was unfx)rtunate. In 1968 both employers .and applicants 
had.been required to list specific areas of competence ^y possessed 
or required for the position. 'In 1969 neither this information nor 
inforrrtation on areas of specialization Was specifically requested on -the 
forms. • Consequently, such infonnation had to b,e inferred from'other 
data requested on the forms. ' 

Applicants were asked to indicate the type of position they were. - 
seeking. It was assumed that peVsons possessed relevant competencies 
In'areas in which positions were sought. However, if the description .of 
the position soughi was. unclear, consideration was given to other infor- 
mation. on the form, such as academic major and previous experience, in 



'This placement service operated during the AERA annual meeting. 
February 6-8. 1969 in Los Angeles. 

^Other data *bn the geographical distributien-*f available positions 
and geqgr'aphical preferences of applicants are reportQ.d later in ttiis 
chapter. ' ' 
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order to construct the clearest description of the area of specialization 
in which the -applicant was qualified -to work. * " 

"Employers were asked to give the title of the position opening. 

c ' m .-■{f t 

a description of the position (including responsi.bilities) , and any 

s'petiaWxperience Requirements/ r 

Al though ^thd'^informati on gathered from both forms was often not 

specific, and althoi#Mthe applicant information was Subject to limitations 

of self- report data, it was fett that comparisons of tITe two sets of data 

9 

might still provide useful information to the Task' Force. 
P rocedures 

The procedures for organizing and analyzing the data are reported 

below. ' ' , 

1. Each employer form was reviewed- and each area^of specialization 

■ ' ' ' '• ' ' » 

that could be inferred from the responses or that was directly stated wa? 

-.' » . 

listed alphabetically. Each different area of specialization was listed 

\ 

separately (e.g., "tests and measurement" was listed separately from 
"measurement"); identical statements were simply tallied. 

2. The resulting listings of areas of specialization required by 
employers were \eol lapsed into the logically derived categories used below 

^ to report the results. ^ - , / 

3. Each employer form was also categorized as to whether the posi- 
tion described was a research, research faclTitative or nonresearch position. 



10 



' ^An assessment of the reTiability of the'self-report data»fronr the 
1970 AERA placement service is' contained In Appendix J, an assessment 
which also has Implications for the 1968 and 1969. data. ' . 

^°Research, research facllftatlve^nd nonresearch. positions were 
defined In footnotes 4 and 5 above. 




4. FlnaUy, employer forms were categorized into ^even institutional 
settings and cross- tabulated against ''areas of specialization." • 

5. Each ^plicant form was' reviewed and each- area of specialization' 
was listed alpNbetically, .using the' same processtas reported ai)ove for 
the employer forms. The resulting lists of areafe of specialization werf 
.collapseij into the same set of categories used' for .the employers. 

6. *^Each application form was also pategori zed as to whether the 
applicant seemed to be seeking a research, research facilitatfv5 or 
nonresearch position. - « . 

. ^ . Results . " . . ' / ^ ^ 

\ Employers registered 459^ vacanci es wUh the 1969 employment serv\(^i: 
569 appircants registered with. the service. Overall, this seems td reflect- 
^ a slight intalance between' supply yd demand."^. However, it is only wteji--"- 
' the vacancies anti alpplications are. analyzed in terms .of areas of special- ^' 
izatioh and institutional settings' that specific supply-demand discrepancie 
can-be' identified. Such analyses are presented in the tables that follow. • 

^^One limitation to. interpreting such supply and deman> data stems"* 
•^^^vfron the fact that, it As impossible to determine precisely what proportion 
• ^ of the applicants occupy continuing positions. For example, 61 Pefceni 

of the applicants were^tudenti evidently completing academic progranw and 
?nneftd bf positions uptn graduation. However, t is ^"H>otj;,Jle ^to determine 
r how many of th.e remaining. 39 percent of the applicants -- J^ose employed ^ 
at the -time when they filled out. the form ~ were holding f ""^"9 Pf^J:. 
tions and looking for new posUioris in the interest of professional advarjce- 
ment, persbnal' considerations, etc., and how many were seeking Positions 
because their positions ^ere phased out (e,§. ,'positions..in discojUnued 
RELs). Therefore., absolute siipply^demand comparisons from data ot Vn s 
type are tewfative at best. If one assulnes, however, that such unceHain- 
. ties in the data are random across areas Of specialization, then relative 
conparlsons of supply/demand discrepancies by area of specialization stiTi 
could be useful. 

' This. same limitation, though not explicated aboV&, Is equally - 
applicable to data l-eported for 196B. . ^ • 
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The tabulation pf 439 of the 'total of-459 vacan#ies,, by'insti- 
\ tutional setting aficLCby a "^etemii nation of whether the v^canc'iTwas for 
a research, research facilitative, or nonresearch position, is shown 
in Taljle 3:6 J^" -^^ ' ' 4 

The tabulation of specific areas of specialization for tho 439 
vacancies for which such informatT&n was available is shown in Table' 3.7. 
'"Note that in tKis^table multiple areas of specialization were listed 

for map of 'the vaci^ncies, resulting in a total number of entries- - 

A ' ' - ^ 

recorded here, far.-in ^,cess, of the-tptal number of vacancies. 

' * • ■ ° ■• ♦ ■ » 

There were 569 applicants in, 1969. 'A comparison of areas of , 

- • - X 
•specialization they listed with specializatiW'listed ^pr vacancies is 

* ' I ' ^ • * 

' presented j^.Table 3.S, tabulated b^ relevance to research and research- 

related areas. "Note again that in this table multiple areas ^oljspecializa 

tion wer6 listed by many emplWrs and appycaiits, resulting in a total 

number of entries 'yffccorded here that is far in excfss of the total^number 

.of applicants or vacancies, 

A comparison. between the number and percentage of vacanci^er that 
" * ,' * ,» ji 

are research,! research facilitative and nonresearch, and the -number of 

applicant* whose areas* of specialization seem to prepare them for such 

positions, is shown in Table 3.9. 

_ ^ , ^ 

''^Owe uniA^eigsity had 20 new vacancies, but did not specify any 
areas of specialization desired; theref&re, those 20 vacancies are not 
^•includSd hefe, . . ' ' 
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Table 3.9 

Numbers and Percentages of Vacancies and Applicants in 1969 
Employment Service: By Ifelevance to Educational Research 
and Research-related Activities (RDDE) 



Vacancy or Applicant 

Areas of Specialization are: 



Vacancies 
N % 



Applicants 
N % 





r. 


Directly Related to RDDE 


249 

* 


56.7 


242 


42.5 


y 


2.' 


Facilitative of RDDE 


64 


14.6 


48 


R.4 


3. * 


Not Related to RDDE 




28.7 


279> 


49.1 






1 

Total 

— \ \ — 


439 


100.0 


. 569 


100.0 



/ 
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til scussion 

Before attempting to derive implications from the data presented' 
above, it should be noted that the same factors that operated to limit 
the interpretability of the 1968 employment service data were also . 
operating here. Despite these limitations, several trends in the data 
appear strong enough* to warrant consideration. 

As in 1968, more than jo percent of the vacancies were in aca- 
demic settings. Although m 1969 applicant form 'did not require 
specific information on the/applicant's institutional setting preference, 
the applicant was asked to' list "position desired." , The majority of 
responses was in the fom bf areas of specialization (e.g. , "evaluation") 
rather than in terms of Institutional preference-^ Of the 126 who did 
specify the type of institution in which they desired employment, only 
nine applicants stated that they would take a position outside of the 
academic setting. Tfus, it would seem that Relatively few persons who 
used this placement service were interested in (or perhaps aware of) 
employment possibilities in RELs, private research agencies and other 
nonacademic settings. » , 

Contrary to the results reported for 1968, where 49 percent 
of the vacancies were related to research or resear^eh-Tffwilitative 
activities, n.3 percent fell in thif^tegory in 1969. Howe^^only 
50.9 percent (290 persorTs)>of the applicants desired such positions, 
as opposed to 84.4 percent (555 persons) in 1968. This would suggest 
. that while research and research ^related vacancies lis^d with AERA 
"stayed almost level, fpi 322 in 1968 to 313 in 1969, far fewer persons 
with relevant preparation were available in the 1969 employment service. 
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Some discrepancies between the number of employees listing 
certain areas of specialization and the number of vacancies in those 
ardas are large enough probably to be indicative of real trends in. 
em|ll5jflncnt markets. Also, some shifts from the 1968 data are notice- 
able. For example, there were more vacancies in research methods in ^ 
1968 than there were qualified applicants to fill them. This was not 
true in 1969; there were slightly more applicants with specialization 
in research methods than^ vacancies in this area. The number of ^ 
vacancies in development increased in 1969, with a balance between 
supply and demand. As in 1968^ no applicant specialized in the area 
of diffusion. ^ ' 

In 1968 there were three openings for eVery applicant' with skill 
in evaluaticJn. In 1969 this discrepancy rose to almost four to one. 
This' lends further support to ■the notion proposed earlier that trends, 
■tow»»"< "c'countability In evaluation and the evaluation mandateslof 
'' Titles I and Ill'of the Elementary and Secondary Education Act of 1965 
will continue to require that more attention be paid to training evalua- 
tion personnel . 

Other areas jn which it appears that far too few people are ^ 
being prepared (in relation to the demand) include applied measurement 
and, to a lesser extent, computer techniques and programming. Con- 
" versely, there seemed to be a marked oversupply of persons prepVred.^ 
(in comparison with relevant vacancies) in guidance' and counseling, 
general administration, curriculum development, and subject matter fields 
However, these latter discrepancies are likely to result, at least in 
part, from the type of clientele using the AERA employment service. 



En?>loyers des1ri?iigr-to hire counse.lors, curriculum directors, and general 
administrators may be more"" inclined to focus their recruitment efforts 
oa conventiojns of professional associations in these areas, whereas 
applicant* may utilize all ava|l*able placement services. ( 

In general, there is -a marj^ed correspondence between the 1968 
and 1969 data. The, need for more persons trained as researchers ger 
se^was greater in 1968 and was reduced in 1969 to something appro)^iina^i4ig 
a balance between applicants and vacancies. However, the need for more 
trained personnel in-evaluation increased in 1969, as ^icL the need for 
pAons trained in applied measurenient. The numbejfc of vacancies in ' ^ 
development also increased to-balance with the supply of persons speciali- 
zing in this ar*ea in 1969. Discrepancies in other areas of specia-liia- 
' tion were not markedly different from those of the previous y^^'r. 
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/ 



An Analysis of 197Q.AERA Employment Service Data ' ^ 
✓< trw)loyment Service Fonns 



The 1970 A^RA" placement service^ ^ used the four forms which 



v 



'^^•are. shown in Afpendik I. The "professional order form" (for emploj^js) 
ami "professionc^l application form" (for applicants) are standard ' 
^^oyment servi.ce forms supplied by the U. *S. Department of Labor. 
These are identical to the forms used in 1968 and similar to those 
from 1969. ' 

' 7 

^ In addition to the two standard forms, an "employer information'^ 
form" (for employers) and an "employ^ information form" (for applicants) 
"were used at the 1970 meeting. These were developed and supplied by 
^ AERA Task Force in an attempt to' gain more specific information_about, 
research competencies required by employers and those possessed by 
applicants. 

Since the two sets of forms (Department of Labor, on the one hand, 

and Task Force, on the other) furnish different types of information, 

, . - I. 

they will bB discussed separately in the following pages. 



Department of L'^bor Forms , 
" The infoTtnatidn-on these forms which ^^<as of particular interest 
for this study was that describiiig areas of competence. Eact>. employer- 
was asked to 11st the competencies required for the available position-, 
and each applicant was asked to give his-^areas of competence. Since no 
specific description was given of the type of response desired, there 
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^■^This placement service operated during March 2-6, 1970 in 
Hinneapolts. 



was -considerable variation in the ]evei of specificity of the responses, 

tn addition, since the information su^ltted by th^ applicants was 

. 14 
self-report data, it was subject to limitations inherent in such data. 

.These limitations notwithstanding, it was felt that comparisons of 

• . •■ * 

applicant and-empldyer data would yield useful information. ' 

^ .' ' , 

. . I : ' 

Procedures 

The procedures for organizing and analyzing the data were as 

follows: , N 

1. Each applicant form was reviewed and each area of competence , 
mentioned was listed alphabetically in the precise manner in' which it", 
appeared on the form. Differing areas of competence were listed 
separately (e.g., "tests and measurement" was fisted separate^ from = 
"measurement") ; identical" areas were simply tallied. 

2. "^The resulting listing of applicants' areas of competence was 
collapsed into theologically derived categories used below to report 
tbo results.^ 

' 3. Each application forni was also categorized as to whether the 
applicant seemed^ be seeking a research position, a research facilita- 
'tive position or a nonresearch jfosition.^^ for this categorization, all 
relevant" information on the form (e.g., areas of competence, special 
information listed, prior professional duties and^areer evolution) was 
used to assist in making a judgment. 

' : ■ ■ ■ 

^^An assessment of the reliability of this self- report data is . 
contained in Appendix J, which concludes that appli.cant responses were 
likely free from the confounding effects of an acquiescence ,set. 

""^ReseaVch, research facilitative and nonresearch positions were 
defined in footnotes 4 and 5 above. ' _ 



4. Each enlployer fomfwas reviewed and each area of competence 

listed alph^tically. using the same process as reported above for 

9ppj1<^t forms. The resulting I'ists of 'areas of competence required by 

era collapsed into the s'ame set of categories used for applicants 
. " 'f : ^- ' 

5. Each employer j'orm was also categorized as to whether the 
position was a research, research facilitative or nonre^earch position. 

s . • \ ^ . , ;■ 

Results N ' .. ' • 

A total of 412 vacancies were lifted witR the 1970 employment" 
service;* 727 applicants also registered. These figures reveal a suL- tan- ■ 
tial inequality between the ^number of positions open^and the number of 
applicants 'for those positions]^ Mor-e specific supply-demand informatioftx^ 
wa^s'obtained^y comparing areas of compete|ce listed, by employers with 
thoseof the applicants. Competencies ^ere tlbulated, by felevancfr j 
to 'research and research- related areas,, and are summarized in Table 3.10 : 
Because-^nultip^competencies were listed for both vacancies and appli-. 



cants, the 



total number of competencies^ecorded fo^-.'&ach groiip is greater 
than the total nuniier of vacanci.es or applicants. (Alt|ogether, 569 .compe- 
tencies were Itsted for 412 vacancies; 1 ,697 competencies were listed by . 

1 

727 applicants.) 



One limitation to interpretinq these figures s terms, from the fact 
that it is impassible to determine piv Isely what proportion of the appli- 
cants occupy continuing positioas. A large percentage of the applicants 
were students ' completing academic progroms and in nee<j of posi tions upon 
graduation. However, it cannot be said how many of the remaining appli- 
cants-'-those employed- at the time when they filled out the fo^J-were 
holding dorvtinuing posffions-and looking for new positions in the interest 

*of professional advancement. personal considerations, etc., and now 
many were seeking positions because their positions were phased out (e..g., 
positions in discontinues! RELs). Therefore absolute supply-demand com- 
parisons from data nf this type are tentative at best. If one assumes, 

•however, that such uncertainties in the data are random across a^eas of 
specialization, then relative compari-sons of supply-demand discrepancieo 
by area of specialization stijl could be useful. 
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Discussion 

r Again, the factors (discussed earlier) that limited the interpre- _ 
tability of the 1968 and 1969 employnient service data applied here. 

In spite of these limitations, some trends are strongly indi- 
cated by the data. First, for all except three categories (survey 
research, vocational education and writing ability), there were more^ 
applicants with specific competencies listed than vacancies in which 
such competencies were relevant. Although this is accounted for, in 
part, by the fact that there were more applicants than vacancies, this 
fact alone cannot explain the size of the discrepancy in a.iiumber of 
categories. While the ratio of applicants to vacancies is 1.76:1, in 
the following areas this proportion is greatly exceeded 

Elementary Statistical Techniques (140-8) 
Advanced Statistical Technigues (24-3) 
General Administration (103-14) 
Educational Research (176-25) 



17.5:1 
8.0:1 
7.4:1 
7.0:1 



17 



Note, however, that the ratio of the total number of competencies 
listed by applicants (1,697) to the number listed by employers (569) is 
2.98:1. The average number of competencies reported by applicants was 
2.3, whereas the average nunter required for positions was 1.4. 
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Guidance and Counseling^ 79- 13) 6-1:1 
Learning/Experimental Psychology (112-19) 5.9:1 
Testing/Applied Measurement, (33-7) ^ 5-^'l 
Research t)esign (93-23) • 4.0:1 

Systems AnJlysis (16-4) ^-O'l ^ 

Contrasted to the data reported earlier for 1968, there now appears 
tc be an overs upply 'of pe;«onnel trained in the areas of statistical tech- 
niques, educational' research and researcT design, testing and applied 
measurement, and systems 'analysis. (As mentioned earlier, apparent over- 
supplies in psychology, counseling and administration may be artifacts 
of the way in which emp'loyers in these areas utilize the AERA employment - 
service.) 

The reversal of trends between 1969 and 1970 is startling; The 
maVket for research and research-re'lated personnel is down in virtually 
all areas, doubtlessly due to reduced funding of critical research pro- 
grams. This has resulted in Jess demand in research, development and 
diffusion than even the least optimistic projections of Clark and Hopkins 
(1969). 

- J£RA T.ask "Force Forms --. - - ■ 

In an attempt to gather ^nore specific information about skills 
required for /^search, development, diffusion and evaluation, the Task 
Force staff constructed a set of employer/employee information forms . 
based on the ski lis identified in the first part of Chapter 2. 
The seven lists of skills presented there were logically 
collapsed into 39 "skills" that were thought to include the most impor- 
tant items from each list. A three-point scale for rating "degree of 
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competence" was attached to each item and the instrument was piloted 
with a convenience sample of 11 persons. The ffnal Instrument (shown in 
Appendix l) was used in both employee and employer information forms. 

All employers and all applicants were furnished with a copy of 
the relevant form and asked to indicate on the three-point scale the 
degree of each of the 39 skills required or possessed. It was ^b^ght 
that comparing responses of employer^ and applicants might provide atidi- 
tional insight into the need for and the availability of important re- * 
search skills. 

i 

Proc edures 

*- 

For each of the 39 items on the information form, the -frequency 
and relative percentage of responses for .the three skill levels were com- 
puted for both the employers and the'employees (hereafter, applicants). 
It was assumed that the skill level indicated by the applicant corresponded 
to the level of skill possesseti. ^ ' 
Two chi-souare analyses were performed as follows: 
(I) An inter-group comparison of skill levels required by 

employers vs skill levels possessed by applicants, 
' (2) An intra-group comparison, of skill levels required by 
Employers for university vs non-university positions. 
Those items for which signfficant values were obtained are given 
in-tabular fonn in the tables which follow?^ (The full tables, including 



""^he skill-items are presented only.by number in Tables 3.11 and 
3.12. The reader may refer to either the employer °^^Jhe employee infor- 
mation fom in Appendix I for a full statement of each skill, in tms 
Ilty Se iw7also nbte those skills for which the chi-square values were ^ 
not significant. 



ERIC 




results for all 39 items, are on file in the Task Force office at the 
Laboratory of Educational Research, University of Cplorado, Boulder.) 
The frequency and relative percentage are reported by group for each . 
"significant" item included. Tables were collapsed for those cases in 
which the expected cell size was five or less. ^ footnote is Included for 
those Items for whi\h the contingency tab.les were collapsed. All analyses 
were perfonned using the Biomedical 02S— Contingency Table Analysis— cfom- 
puter program. 

Results and Discussion 

Although Task Force Information fonps were provided to all em- 
ployers and applicants who registered with the 197D AERA employment 
service, a large number of registrants did not complete them. Forms 
were returned by 81 of the 412 employers and by 361 of the 727 appli- 
19 

cants • 

In the sections whfch follow, for each ^analysis the discussion 
is organized around the major functions of research, research-based 
development, diffusion and evaluation. 

Comparison between Employer and Applicant ''\ 

The Tesp'onsBs- of employers" and applicants were compared by 

Computing a between-groups chi square for each item. These analyses are 

summarized, for the significant items, in Table 3.11. 

l^his represents an applicant response rate of 50 percent and an 
employer response rate of 20 percent. In the absence of sufficient in- 
formation to complete an adequate non- response check, it could not be 
determined what factors may have resulted in differential responding. 
Therefore, the fact that these responses may be biased in some unknown 
way should be kept in mind when interpreting the results presented herein 
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> '■■ \ Table 3.11 

Compart son\of Level of Skill. Required for 81 Vacancies and 
LevelXof Skill Listed by 361 Applicants in the 
\ 1970 AERA Employment Service 



Tfpm 


Degree of Skill 


X 


• 1 

1 

OT 


n 
M 

i 


No 


So^iewhat 


Highly 


. f . % 


f \ % 


f % 


' .1. 


5 0. b 
5 1.4 


AA \^7 1 

149 41.5 


2 8 36 4 
205 57.1 


10. ?6 


1^ 


•c.Ol 


^ x 


19 23.7 
27 7.5 


A^ Ki 9 
200 55.7 


?n 25 0 

132" 36. ar 


19.3 


2 


-.O'l 




3 3. o 
5 1.4 


12 3 34.4 


36 46 2 
*230 64.2 


8.81 


r 


«:.01 


I 


19 23.7 
63 • 17.7^ 


CD 0 1 . O 

199 55.9 • 


36 45 0 
94 26.4 


16.57 


2 


'.01 


13 ^ 


5-' 6.3 
20'" "5.6 


25' 31 . 3 
168 47.1 


50 62.5 
'169 47.3 


6.01 


r 


■ <.05 


} 11 


. 14 17.7 
55 15.4 


27 34.2 
203 56.7 


38 48.1 
100 27.9 


14. 80 


o 
C 


^ m 

' . U 1 




11 ' 13.9 
37 10.3 


23 29.1 
202 56.4 


45 57.0 
119 33.2 


19.84 


2 


<.01 




18 23.1 
100 28.6 


21 26.9 
192 54.9 


39 50.0 
58 16.6 . 


42.15 


2 


s.Ol 




24 31.2 

T33~ 37.5' 


30 39 .0 

7 173 4S^.I: 


23 29.9 

4^ „ ~^a^. 8- 


.J1.7.8_. 






34 A 


2 4 30,4 
' 65 18.5 


31 39.2 
195. 55.6 


24 ^30. 4 
91 25.9. 


8.13 


2 


V.05 



*Chi square computed from collapsed table obtained by -c'ombining "no ski 11" . 
"somewhat skilled" levels. 
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Research ; Significant differences between the employer's 
requirement for a specifTc'skil 1 and tfje applicant's possession of that ^ 
skill were foynd in only three cases. For items 1, 2. and 4 the skill 
level possessed by the applicant was significantly greater than that 
required by the employer. Items 1 and 4 relate to the 'identification and 
delineation of a research able problem-. The significance achieved on 
item 2 may indicate that few empToyers were looking for people to work in . 
research-administrative positions; thus such skills were not highly rated 
by the employer group. =No significant differences were f^nd for the 
remaining eigh,t research -related Items. For these items, both the skill 
required by the employers and the skill possessed by the applicants. we re ' / . 
rated somewhat high, with only a' small proportion indicating no.skjll. 

Re search-based Development ; Five of the seven items related to 
research-based development yielded significant chi-square values. For ^ 
>temsJ2, >3,.14,'15, and 16, most employers requ1red-a high skill' level ' 
whereas most applicants were only somewhat skilled. It is noteworthy 
that the significance level of item- 13 is lower than the others (.05 vs ( 
.01). This observation may suggest that evaluation-related skills within 
research-based development are not .as well delineated as those skills 
-more' closely related to the developmentaT xispects of in^-trtictiwtal-sys terns-.' 
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Setting the alpha level at p=.05 may result in some spuriously 
significant results due to the large number of chi tests run. Given the 
exploratory nature of these analyses, however, it w^is thought desirable 
to identify any differences which mightjexist. 
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Diffusion s Only one item yielded a significant chi square 

' between the employer's requirement of a specif ic«ski 11 and the appli- 

cant!s. possession- of that skill. For item 20, the'level required by 

f.§ the 'Employer was significantly greater than that possessed by' the appli- 

^ cant, (However, the greater 'proportion of both groups indicated no ski^l 

to moderate skill.) No significant differences were found for the re- 

maining seven items related to diffusion, • . . 

Evaluation: Of the thirteen items related to. evaluation, only 

item, 34' yielded a significant chi-sc^are value.. For this item,^ related , . 

to process evaluation, the greatest percentage of appli ca^its, rated them- 

^selves as somewhat skil'led whereas the'^ degree of skill required by 

enployers was more even.ly distributed across all three skill levels. 

. . ■•■ ' . 

Comparison between University and Non-Universi ty Positions , ^ 

The skill levels -required by uni.Versity-based employers and by 
non-university-based employers were compared by computing a between- groups 
Chi square for each item. For the purposes of this analysis, positions 
available in universities or R & D centers were categorized as university 
positions; positions avai-lable in regional laboratories, independent 
— 'research^ agencres-, schxjal districts, state -education-departmentsr-federai— 
* agencies, the military and industry were categorized Ss non-universi,ty 

positions. The analyses for those items for which significant differences 
- were found are summarized in Table 3.12. ' 

' Research ; Significant differences between the skill level 
required for universi-ty positions and that required for non-university 
positions were found in only, two items. For items *2 and 8 the non- 
university level was significantly. greater than that required forXiniver- 
sity positions. The absence /of administrative positions at the university 
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. ' . Table 3,12 .. 

cdmparison of Levejs of Skill Required for University- 
j based and Jijoir^iversi'ty-based Vacancies in the 
. • 1970- AERA Employment Service 



Item 

i 


Degree of Skill 


\ 

1 -.■ \ 


r df 


- 

p ' 


No 

Skill 


Somewhat 
Skilled 


Highly 
Skilled 


, - f ., « 


f * % ; 


f .% ., ' 


2 U 
. NU 


' 14 24.1 
. 5 - 22.7 

' * - 


36 62.1' 
5 22.7 


8 13.8 
12 54.5 


15.43' 


\;2 


<.01 


^ NU 


5 8.6 ' 
• .0 ^ 0.0 


■ — ' — 

31 . 53.4 
4 19.0 


' 22 37.9 
« 17 81.0 


11.42 . 


\ 

1«\ 


<.01 

\ 


17 ' " 
" NU 


- 9 16.4 
r 4.5 


32. 58.2 
10 -45.5 


14 25.5 
11 50.0 


4.32 


' 1^' 


\o5 

■ \ 


NU 


22.. 39.3 
A 1Q n 


24 42.9 


10 17.9 
10 47.6 


7-. 40 


2 


<,05^ 


2d fttH 

NU 


14 25.5 
. 1 4.8 


31 56.4 
11 52.4 


10 18.2 
9 42.9' 


' 4.94^ 


1^ 

e 


<.05 


29 " 
^ NU 


15 26.8 • 
1 4.8 




33 58.9 
5 23.8 


8 14.3 ' 
15 71.4 

j 


23.81 


1^ 


<.01 


30/ " 
"*^\ NU. 


13 23.2 
1 4.8 


22 39.3 
5 23.8 


21 37.5 
15 71.4 


7.06 . 


1« 


<.01 


31 " 
^' NU 


28 " .50.9 
3 14.3 


■ 23 ' 41-8 
8 38.1 


4 7.3 
K) 47.6 


8.44 


l'' 


<.01 


36 ,|i 


7 12.7 

_i 4.-8 ^ 


30 54.5 

4_ _ ...1^0„- 


18 32.7 • 

— 1 6 — '-76 » 2" - 


11. '61 


1 


.<.01 














37 " 
NU 


5 8.8 
0 0.0 


25 43.9 
3 14.3 


27 47.4 
18 85.7 




9.24 


1^ 

-\ ■— 


<.01 


^ NU 


8 14.0 

0 0.0 " 


27 47.4 
6 28.6 . 


22 38.6 
15 . 47.4 ■ 


6^3 




<.05 


39 ^ 
''^ NU 


5 8.8 
1 4.8 


32 56.1 
3 14.3 


' 20 35.1 
17 81.0 


12.95 ■ 




<.01 
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^Chi square coirpu^d from collapsed table obtained by combining "no s-kill" 
and "somewhat skilled" levels. 

''Chi square computed from collapsed table obtained by combining "somewhat 
. skilled" and "highly skilled" levels. 



level accounts for this discrepancy for item 2; no explanation is readily 
apparent for the discrepancy on item 8. No significant differences were^ 
found for the remaining nine research-related items in this comparison. 
For/ these items the skill required for university positions and non- 
untVersity positions was rated moderate to high, with only' a small pro- 

portion requiring no skill. 

Re search-based Development : Cmiy one development item yielded 
a significant Chi square Wweeti the skill level required for university 
positions and that required for rfon-university positions . For item J7, 
the non-university positions requirecj.ja higher degree of ski IV than uniy^r 
sity positions. Thjs difference may 'be attributed" to the composition of 

. the non-university group, which includes positions at state educational 
agencies, school districts §nd federal agencies, all of which have con-x^ 
siderable contact with school^ and classrooms and reflect a more practical 

.than theoretical emphasis.' No significant differences were found for the 
remaining items related to research-based development. For thesejtems, 
both university pos-itions' and nonruniversity positions required moderate 

to high skill. ; .- , ' ' , 

Diffusidn : Item 26 yielded a significant chfrsqu'are value ■ 

revealing that the sk'ill level' required for university positions was ^ 

lower than that for non-university positions. This ^fference may be . 

attributable to the practical emphasis' of the n on -university group. No ; 

significant differences were found for the oljher diffusion-related ttente^. 
Evaluation: • Significant differences i)et\5»een''th9 university 

ein)loyer's requirement of a specific skill and the non-university - ■ 



■f- \ ■ ■ ■ 

^employer's requirement of that skill were found in eight cases, Fdv 
■ items 28,^9, 30 and 31, each related to contexf^evaluation. and for ' 
items 36, 37, 38 and 39j each related to outcome ^Valuation, the skill 
level required for non-uni versity positions was s.ignificantly greater 
than^at required for university positions. J-his may result from the . 
greater concern with "public schools" mani fested. by members of the non- 
universUy group. No sij^iificant differ^ces were fo)ind for the items 
related to program planning and procfess evaTttation. 

Concl usions 

■ , ' 

It was stat2 earlier that, it "'would be presumptuous to attempt 
to draw firm conclusions from the data presented in thi^ paper." Nonethe- 
less, the above discussions of results from the two sets of fonj^ may 
suggest the following:/ , ^ ' 

' n.. At present, there appears to be an .versupply of persons 
drained in educational research and research .designT stati'stical tech- 
niques, testing and ap'plied 'measurement, 'and systems analysis. 

2. The demand for persons trained in development and diffusion 

which was predicted by .Clark and Hopkins (1969), is riot*>ow evident in 

»' ■' ' • •. 

the AERA employment service. 

3. ' The "no'skill" response on the Task F«rce, forms was ,cHosen ' 
more often— uy both employers and applicants-ron those items not^ related 
to research r-: to outcome evaluation. This may indicate that develop- 



^Vhe "no skill" response on research and outcome evaluation 
items was about 8 percent for employers and 5 percent for applicants. 
On development, diffusion, and formative evaluation items, the no 
skin" response was about 23 percent for employers and 22 ^)ercent for 
applicants. ' * . 'v 
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ment, diffusion and formative evaluation'-TRills 'are not well defined, 
.or it m^y suggest further (see 2 above) that such skills are not in 
great demand. 

4. Evaluation skills are required in a higher degree for 
non-university positions than for those inside the university. 
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' ^^'Seventy^eight percent of the university employer responses 
on evaluation items required either some skill (49 percent) or a high 
degree of skill (29' percent). The comparable figures for the non-uni- 
versity group are 33 percent for some skill, 59»,percent for a high 
degree of skill-a total of 92 percent. 
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Geographic Distribution of Positions and Applican ts. 1968-1970 

The "professional application 'forms" used in the AERA employ- 
ment services in 1968, 1969 and 1970 all included an Item on which 
the applicant could list any preference he had for geographical 
location of the job he was seeking. At the same time, the 'geographi- 
cal location of each position registered with the employment service 
was available from the address required on the "professional order 
forms." It was thought that comparing the locations of jobs with 
locations preferred by applicants would show whether there were 
obvious supply-demand imbalances in particular geographic areas. Such 
findings could reveal trends in geographic job mobility that might have 
impllcatidns for the geographical distribution of new research traim^ng 
programs. Therefore, a tabulation of geographic data from the forms 
was conducted to see if such trends existed. 



Procedures 

The location of each position registered with the employment 
service was placed in one of eight regional categories as shown in 
Figures. 1. 

ThV "geographic preference" date from the employee application 
fonns was Xinitialli^opied down in the^ exact manner in which it 
appeared onNthe form. Responses were 'later collapsed intp t4ie same 
categories isvthose listed in Figure 3.1. However, many applicants 
stated "no preference" or "any location," and some applicants omitted 
the item completely. "Omits" were counted as a1?o, having no preference. 

Single prefere\ces (e.g. , i!M1dwest," "Florida," or "Boston") 
' could generally be ca^gori zed accurately, except for those that wer^ 
either nonspecific or niqnregional* in character (e.g., "U. S. A.," 



\ 



23, 



\ 



^Copies of the application and order forms used In 196ff, 1969 
and 1970 will be found in Appendices G, H and I. 
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New England 

Connecticut 
Maine 

Massachusetts 
New Hampshire 
Rhode Island 
Vermont 

Middle Atlantic States 

Delaware' * \ 
Maryland^ 
New Jersey 
New York 
Pennsylvania 
. Washington^ D.C. 

South 



Alabama 

Arkansas 

Florida 

Georgia 

Kentucky 

Louisiana 

Mississippi 

North'^ Carolina 

South Carolina 

Tennessee 

Virginia 

West Virginia 



Scfuthwest 

Arizona 
New Mexico 
Oklahoma 
Texas 

Mountain States 

Colorado 

Idaho 

Montana 

Nevada 

Utah 

Wyoming 

Pacific/West Coast 

Alaska " 
- California 
Hawaii 
Oregbn 
Washington 

Canada 



Midwest 



Illinois 

Indiana 

Iowa 

Kansas 

Michigan 

Minnesota 

Missouri 

Nebraska 

North Dakota 

Ohio 

South Dakota 
Wisconsin 



Figure 3 J: States Included in Each Region 
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"warm climate," "urban area," "near husband^s job," "abroad," Non- 
specific and. nonregional responses were assigned to the category of 
"other," Multiple preferences were more difficult. Where the multiple 
preferences were in the same region, the response was simply tallied in 
that regional category. However, when multiple preferences were in 

different regions (e,g,, "Midwest or West Coast") one tally was assigned 

24 25 

to each of the two regional categories, ' 



Geographical Location of Jobs and Preferences in 1968 

Tabulations of available positions and geographical preferences of 
applicants for 19618 are summarized in the following three tables. Fre- 
quencies and percentages are given for each region; positions are also 
broken down by the type of institution in which they are located. 



^he. inflation in number of appat^nt applicants was not viewed as 
a problem here. Comparisons of national supply-demand discrepancies, 
in actual numbers, have been reported above. Here the interest is in 
the relative number of applicants receptive to employment possilities 
in each region* 

^^here may be some Inaccuracies in assigning responses to categories 
since it is impossible to know whether respondents intended the same 
area where they used referents (e.g., "Midwest") as that area listed ^ 
under the same referent in Figure 3.1, Also, assumptions were made in 
assigning the relatively small number of responses that could not readily 
be fit into the categories used in this papier. For example, persons 
who indicated a preference for the "East Coast" were arbitrarily placed 
in the "Middle Atlantic States" category* While this is likely to be 
a correct categorization in most instances, it probably results in a 
slight underestimate in the "New England" and "South" categories. How- 
ever, relatively few assumptions of this type are reflected in the data' 
reported here. 
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Table 3.14 

Locations of 769 Vacancies Listed in the 1968 AERA Employment 
Service: By Regions and States 



New England 

Connecticut 
Maine 

Massachusetts 
New Hampshire 
Rhode Islands- 
Vermont 

Middle Atlantic State 
OeHware 
Maryland 
New Jersey 
New York 
Pennsylvania 
Washington, D.C. 

South 



Alabama 
Arkansas 
, Florida 
Georgia 
Kentucky 
Louisiana 
Mississippi 
NortVt Carolina 
Sou tiy Carolina 
Tennessee 
Virginia 
West Virginia 



2 
5 

14 

1 
1 

6' 



0 
9 

10 
101 
85 
77 



0 
0 
IC 
3 

14 

2 
2 

10 
1 
7 
5 

34 



/ 



Midwest 

Illinois 

Indiana 

lOMa 

Kansas 

Michigan 

Minnesota 

Missouri 

Nebraska 

North Dakota 

Ohio 

South Dakota 
Wisconsin 

Southwest 
Ari zona » 
New Mexico 
Oklahoma • 
Texas 

Mountain States 

Colorado 
Idahij.--^ 

Montana 

Nevada 

Utab 

Wyoming 



70 
48 
5 
6 
28 
17* 
35 
1 
2 

#25 
3 

12 



2 
5 
0 
12 



0 

0 
0 
2 
0 



(Table Continued) 
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Table 3.14 (Continued) * 



Pacific/ West Coast 

' Alaska 
California 
Hawai i 
Oregon 
Washington 



Canada 52 



0 
15 
9 
7 
9 



/ 
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Table 3.15 

'GepgraphicJil Preferences of 776 Applicants in the 
* 1968 AERA Employitient Service 



Geographical Area 




% 


No Preference 


350 


41 


New England . 


24 ' 


■ 3 


Middle Atlantic States 


104 


12 


South 


37 ; 


4 


Midwest 


72. 


9 


Sogthwest 


12 


1.5 


Rocky Mountain States 


29 


3.5 


Pacific/West Coast 


101 


12 


Other 


120 


14 


* 

Total 

> •* 


849 


100 



Note: The total exceeds 776 due to multiple pre- 
ferences listed by seme applicants. 
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since 350 of the 776 applicants^stated no geographic preference 
whatever, and since 14 pfercent of the responses were for "other" locations 
which could not be categorized. It is difficult to make meaningful compari- 
sons between the vacancy and applicant data. Nonetheless, some apparent ^ 
intalance appears between Tables 3.13 and 3.15. • For example, iri the Middle 
Atlantic States there were 2.5 vacancies for every applicant who stated 
a preference for that area. The ratio in the South was about 2 to 1, 
in the Midwest about 3.5 to 1. In contrast, the Mountain States region 

If 

appeared to attract almost 5 applicants for each opening, while the 
Pacific/West Coast drew '2.5 applicants for each vacancy. Note also the 
considerable number of openings in Canada, with no apparent applicant pre- 
ferences for that area. 

Geographical Location of Jobs and Preferences in 1969 

The sunmary of geographical data gathered from the 1969 AERA place- 
ment service employer and applicant'' forms is presented in Tables 3.16, 3.17 

And 3.18. ■ ^ 

In the 1969 employment service, 351 out of 569 applicants stated no 
preference for the geographic location. of the jobs they sought. Apparently 
many applicants were unconcerned about the area in which they were located; 
as some of them said, "The type of job is more important than the location. 
For those who did express a preference, however,' some statements may be 
made about their preferences in relation to actual job openings. 

FiT^t, there was more overall supply-demand balance than had been 
observed in 1968. Secondly, however, a' gross imbalance existed in the 
South (13 vacancies for each applicant preference) and in^the Midwest (a 
' ratio of 5 to 1), In addition, there were more 'applicant preferences in 
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Table 3.17 

Locations of 459 Vacancies Listed tn the 19'69 AERA 
' Employment Service: By Regions and States 



New England 




■ III ^ 

" Midwest 




Connecticut 


U 


1111 noi5 ^ 




Maine ^ 


U 


i noi ana 


30 


Massachusetts 


o 

8 


Iowa 




New Hampshire 


0 


Kansas 


8 


Rhode Island 


4 


Michigan 


7 
/ 

i 


Vemiont 


0 


Minnesota 


5 




• 


^ Missouri 




Middle Atlantic States 




Nebraska 


u 


Delaware 


0 


Nortn uaKota 




Maryland 


9 


Ohio 


,18 


New Jersey 


on 


boutn uaKOta 


Q 


New York 


52 


V(jsconsin - 


O 

0 


Pennsylvania 
Washin^on, D.C. 


19 
6 


\ • . 
Southwest 




ft 




Ari zona 


^ , 


South , 




New Mexipo^ 


** 


Alaban^ 




UK ianoma 


n 


Arkansas 


u 


Tovac 
1 CAOO 

ft ' _ 


15 


Florida 
Georgia < 


15 
2 


/ 

Mountain States 




Kentucky 


13 


Colorado 


3 


Louisiana 


0 


Idaho 


0 ^ 


Mississippi 


. 1 


Montana 


0 


North Carolina 


1 


Nevada / 


5 


South Carolina 


6 

♦ 


^ Utah 


\ 


Tennessee 


8 


Wyoming 


0 


Virginia 


4 






Wes\ Virginia 


.2 







O ' ' (Table Continued) 
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Table 3.17 (Continued) 



Pacific/Wes t Coast 

Alaska 
California 
Hawai i 
' Oregon 
Washington 



0" 
45 

0 , 
.14 

6 



Canada 



Other 



28 



• I * 



y .1 



% ■ 



Tablfi3.18 



1 t 

Geographical Preferences 'of 569 Applicants in the 
* ' 1069^RA Employment Service 



Geographical Area. • ^ . * 

No Preference 351 \ ' 55 * 

JNew England 12 2 . 

Middle Atlantic States "95 15 

South ■ ' 4 0.5 

Midwest 33 , 5 

-s 

Souj:hwest • ■ . . . ^ * " ^ 

Mountain States ' 10 ,1.5 

Pacific/West Coast • i ' 

Other " . • ' j 30 ' ■ ,5 

\ 

Total .. " •638,- 100 



Note: The total exceeds 569 due .to multiple pre- 
< ferences listed by some applicants. , \ 





... 


* * 
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the Pacific/West Coast region than there were openings, although the 
imbalance was not great. " Again note that no appl icant. expresse£ a prefer- 
ence for Canada. ' ! 

Geographical Location of Jobs -and Preferences in 1970 

The suimaries of geographical data gatfi&red from the 1970 AERA place- 
ment service forms are presented in Tables 3.15, 3.20 and 3.21. 

In 19.70 a^ in previous years, *a large number, of applicants (405 out 
of 727) did .not state a preference for the geographic location of their 
jobs. Among those who did, however, some observations may be made. The 
South, had 3J5 openings for ever^ applicant who preferred a job in that — . 
region. In the Midwest the ratio was a little less than 3 to 1 , in the 
Southwest about 2 to 1. The Mountai n States , en the other hand, had far 
more applicants than vacancies (about JO tal), as did the Pacific/West 
Coast (aboiJt 2 to 1). For the first time, a small number of applicants in- 
'dicated a speei fie, preference for Canada. 

Conclusion 

Table 3.22 Was prepared in order to facilitate examination of data for 
the three years covered, in this paper. For each year, the frequency and 
percentage of job openings and applicant preferences are listed for each 
region; for appl icants ,- the "no preference" figures are aJ.so given. 

'■ The most conspicuous fact which emerges from this table--though pro- 
bably not the most surprisi ng-is the large number of "no preference" indi- 
cations by employment service applicants. There is no easy-way, of course, 
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Location of 412 Vacancies Li 
Employment Service: By 



Table 3.20 

sted in the 1970 AERA 
Regions and States 



New England 
Connecticut 
Maine 

Massachusetts 
New Hampshire 
Rhode Island 
Vermont 

Middle Atlantic States 



Delaware 
Maryland 
New Jersey 
New York 
Pennsylvania 
Washington, "* 

South 

Alabama 

Arkansas 

Florida 

Georgia 
^'^(entucky 

Louisiana 

Mississippi' ' 

North Carolina , 

4 

South Carolina 
Tennessee 
yir.gfnia- 
West Virginia 



1 
7 

15 
0 
8 
0 



2 
4 
l'4 
40 
10 
10 



4 

2 
4 
8 
9 
. 0 
0 

13 
4 

2 

8 , 
2 



r 



Midwest 
Illinois 
Indiana 
Iowa 
Kansas 

Michigan f 
Minnesota 
Missouri 
Nebraska 
North Dakota 
.Ohio 

South Dakota 
Wisconsin 

Southwest 
Arizona 
New Mexico 
Oklahoma 
Texas 

Mountain States 
Colorado 
Idaho 
Montana 
Nevada ^ 
Utah 
Wyomi ng 



\ '50 
'11 
2 
3 
12 
25 
♦2 
0 
0 
18 
5 

14 



1 
1 

6 

18 



2 
0 
0 

0 \ 
0 ' 

0 



J 



(Table Continued) , 
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Pacific/ West Coast 
Alaska 
California ' 
Hawaii 
Oregon 
Washington 



Tab)/ 3.20 (Continued) 



// 



0 

43 

1 

0. 
7 



Canada 



Other- 



21 
3 
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Table 3.21 

Geogr*aphical Preferences of 727 Applicants in the 
1970 AERA Employment Service 



beograpnicat area , 




% 

to 


No Preference 


405 


52.5 


New England 


26 


3 


Middle Atlantic States 


78 


10 


South^ 


16 


2 


Midwest 


54 


7 


Southwest 


11 


1.5 


Mountain States 


21 


3 


Pacific/West Coast ^ 


98 


13 


Canada 


■ 4 


0.5 


Other 


59 


7.5 


Total 


772 


100 



Note: The total exceeds 727 due to multiple pre- 
ferences listed by some applicants. 
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to determine whether these persons, in fact, have no geographic preference 
regarding' job location. Some applicants may feel that by stating a pre- 
ference for a specific area they will exclude themselves from consideration 
for openings outside that area. For others, geographic location may be an 
important criterion if they have more than one job offer to consider, (It 
may be more important than salary level, for example.) In any case, it is 
difficult to interpret these "no preference" data with any real degree of 
assurance. 

An examination of Table 3.22, by region, discloses some interesting 
trends?^ First, there is a fairly good balance between supply and demand in 
New England across all three years. Second, the large disparity between 
vacancies and applicants in the Middle Atlantic States in 1968 (more than 
2.5 vacancies for each applicant) completely disappeared by 1970. Third, 
there is a marked imbalance in the South for all three years, with many more 
vacancies listed than applicant preferences. This is also true in the 
Midwest, where a large number of openings is typically avaifable. The South- 
west also shows more vacancies than applicant preferences across the years, 
but in this case the numbers involved are relatively small. Fourth, only 



All statements about regional supply-demand trends are true for 
persons using the AERA employment service. While it seems reasonable to 
assume that these trends are representative of other vacancies and appli- 
cants in research-related areas, there is no good way to know if this is so. 

An additional caution should be kept in mind in interpreting these 
statements. The fact that roughly every second applicant seems open to 
recruitment in any region suggests that the Tow applicant interest in some 
geographical regions may not be a problem; the positions could be filled ^ 
from those who express no preference. 



168 

the Mountain States and the Pacific/West Coast region show more applicant 
preferences than vacancies, and in both cases this is true for all three 
years. Finally, in all three years there are considerable "other'* appli- 
cant preferences, as well as nunerous vacancies in Canada' for which few or 
no applicant preferenceb are expressed. 

It is doubtful whether any significance can be attached to the last 
two observations. The ''other" listings, because of their nonspecific and/ 
or nonregional character, defy inter;>retation. In the case of the Canadian 
vacancies, it is felt that the near absence of applicant preferences is, 
for the most part, the result of oversight. 

Some consideration must be given, however, to the situation in the 
Midwest and South, on the one hand, and in the Pacific/West Coast region, 
on the other. Because of the number of vacancies and applicants involved, 
and because of the consistency and size of the supply-demand imbalance, the 
job market in these regions deserves wider and more critical study than 
has been possible here. 
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' Implications for Supaly and Demand and Emphases 

in Research and Res e a rcti- related Roles 
— — ^ 

Introduction 

, In previous sections of this report, the importance 9f current 
data on the supp-ly and demand of educational research and research- 
related personnel has been elaborated. The primary purpose of this 
section is to examine four of the previous analyses and to look for 
correspondences and trends across time that are contained therein. 
Specifically, the 1969-70 'telephone interviews of er^jloyers (Chapter 2) 
as well as the data from the 1968, 1969 and 1970 AERA employment services 
will be considered. The task is undertaljien^by first' summarizing brief/y 
for the convenience of the reader the analyses from which data are ^awn 
and reiterating limitations of these analyses. Subsequently, the data are 
interpreted in tepms of general supply and demand trends and also in 
relation to changes over time in the particul-ar skills required by 
employers compared with those skills reportedly possessed 'by applicants. 
Finally, a few salient Implications are derived that seem to be substantiated 
by .the data; 

Analyses Providing the Data Base 

In the 1969-70 telephone interviews, 60 employers from 10 institutional 
settings were contacted and asked questions on the relative importance of 
certain research, development, diffusion and evaluation (RDDE) skills. 
The 60 respondents reacted to a moderately extensive list of skills covering 

seven functional areas (research"^ research-based development,' diffusion, 

/ ' ■ . . 

context evaluation, program planning/input analysis, process .evaluation, 



and outcome evaluation). Employers responded to the list of skills In each 

of the seven functional areas by (1) suggesting whether or not the list 

■ • ' o 

of skills was Inclusive^ (2) indicating those skills within each. function 

that were inipj>rtant in the respondent's line of endeavor, (3) Indicating 
those skills that were unimportant, (4) indicating new skills that hid not 
b^en included on the list, (5) identifying skills that were "hard to come by" 
and in short supply, (6) indicating their knowledge of existing training 
programs dlsigned to develop in trainees the "hard to come by" skills 
(as well as sites where such skills^were being developed Incidental to other 
activities), (7) describing new techniques. or methods for training personnel 
in the relevant short-supply skills, and (8) suggesting skills which the ^ 
enfjloyers felt would become necessary in their research programs at some 
future point in time. Fifty-eight of the 60 Interviews were usable 'and 
data generated therein were reported in detail in Chapter^. 

The three studies concerning the AERA employment service are varipd, 
partly because of variations in the employment forms used each year and 
partly by design, particularly in the case of the 1970 employment service. 
In the 1968 and 1969 AERA employment services, data were available on 
areas of competence reported by applicants and areas of competence called 
for by employers, although considerable analysis and categorization were 
necessary. The basic strategy, as reported earlier in this chapter, was 
to compare the self-reported competencies of the applicants with the compe- 
tencies required for the vacancies listed by employers. In the 1970 AERA 
employment* service, additional information was obtained from both employers 
and applicants via forms designed by the Task Force. The information 
forms for enpldyees requested that the applicant indicate whether he was 
h1gh:iy skilled, somewhat skilled, or unskilled in 39 specific competencies 



171 

• ■ 

! * 

in research, evaluation, development and diffusion. Employers indicated 
on their information sheet whether a person highly skilled, somewhat 
skilled, or unskilled ( for each of the 39 listed competencies) was needed 
to fill thepositions for which they were seeking applicants. The details 
and outcomes of each study are reported in Chapter 2 and in earlier parts 
of this chapter. 

L imitations of the Data-Based Studies . ° ^ 

♦ 

It was pointed out In the previous discussions that there are several 
inherent limitations in the data which provide the base for this study. , 
These limitations are brief|^suninar1zed in order to emphasize certain 
inadequacies of the data that are relevant to interpretations made here. 

1. One major limitation is that all the data from applicants (and 
employers,, for thfft mattet^ is substantially self-report data;. the possible 
biases that can appear in data collected /through self- reports are well 
documented and will not be re-examined here. . .« 

2. A second limiting factor is that no detailed description of 
desired response formats was provided to either empToyers or applicants; 
therefore, there was considerable" variation in the degree of specificity 
of the responses with many of the responses quite global in nature. 

3. A third limitation is that there is no way to ^cnow for certain 
whether employers and applicants used the same referents in listing areas 
of competence. Distortion likely resulted from this but it Is difficult 
to- estimate the degree to which it occurred. 

4. Another difficulty with the data resides in the fact that it 
is not possible to determine precisely what proportion of applicants 

ERIC 
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occupied cont^inuing positions. Although the greatest percentage of appli- 
cants were students completing advanced degrees, the ser*vice .obviously 
was used by many other individuals as well. 

5. The shift to different employment service forms in 1969 
'seriously reduced the comparability of the data obtained over time. 

6. There is a lack of knowledge about (a) who uses the ^employment 
service. and (b> variations that might be expected 'in numbers and types of 
user as the convention site moved from city to city. (Data will be pfresented 
later to document- large variations tha,t occurred in many categories when 

the site moved from Chicago in 196Q to Los Angeles in 1969 to Minneapolis 
in 1970.) 

7. A final limitation that should be noted is the difficulty inj^ 
determining the orientation of the employers using the AERA employment 
service or shifts in their orientation over time. It Is assuhieq ihat for 
many of the ^reas of competence listed in the tables below employers would 
be likely to recruit'prospective employees at conventions other than AERA. » 

I 

Trend s in Sapply and Demand 

In spJte of the limitations of the data enumerated above, it is 
felt, that there Is merit. in examining the data in an attempt to deterfiiine 
general trends in supply and demand. The number of applicants and the 
number of vacancies listed by employers in 1968, 1969 and 1970 ar§ indicated 
in Table 3,23; in addition, ^he total number of vacancies and applicants is 
graphically displayed in Figure 3.2 in order to facilitate examination 
of trends. 
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It is apparent in the table and the figure that the nuirber of 
vacancies listed with the placement service fell dramatically in the 
25-mortth period from February 1968 to March 1970 (from 769 to 459 to 412). 
The nunter of applicants, on the other hand, fell in 1969 (from 811 to 569) 
but then returned in 1970 ti 727-a level comparable to that of 1968. 

In attempts to inteipret the sharp reduction from 1968 to 1969 in 
both applicants and vacancies, it was noted that the percentage reduction 
was greater for vacancies. Also, the rate of vacancy reduction was much 
more pronounced in the 1968-1969 comparison than in the 1969-1970 
comparison. Although it is not possible to be certain, one factor which 
quite likely influenced the reduction is the site selected for the con- 
vention." AERA draws larger proportions of its membership from the upper 
Midwest than from other sections of the country. In 1968 Chicago hosted 
the" AERA convention --as it h,d done in most previous years--and 4,509 
persons attended. The 19'/9 annual meeting was held on the West Coast 
"tor the first time in AERA's history and attendance at Los Angelus fell 
to 3,600^ Additionally, it is probable that members from the Midwest 
and East were those less able to, attend because of time and cost factors. 
This restriction was probably particularly true for graduate students 

(other than those going to, school in California), the largest single 

27 

group normally appearing as applicants in the placement serv ce. On 
the .other hand, one would not expect that persons recruiting new employees 

' i 
— <ii 

27it should be noted, however, that some applicants are listed 
with the employment service even though they do not attend the AERA 
convention« , - ■ ' . ^ 

, ■ • ' 



would be as significantly restricted in attending the California meeting 
if personnel vacancies existed and recruiting was underway. 

It would appear then that reductions in vacancies and applicants 
from 1968 to 1969 can be attributed in part to the site selected. Further, 
the reduction in applicants probably was more affected by the site and 
therefore probably is less indicative of a true decline than is the case 
for the vacanci'es. Employment recruiters presumably woul? be less 
restricted in attending a convention because of financial consideratidqs; 
therefore the large percentage reduction In vacancies must arise from 
factors other than convention location alone. 

The discrepancy between vacancies and applicants increases 
markedly from 1969 to 1970 (Figure 3.2). It should be noted that the 
attendance at the March 1970 annual meeting in Minneapolis was even lower 
than at Califohiia the previous year (3.400 compared to 3.600). Despite 
the reduced attendance, however.', the number of applicants at the place- 
ment service increased substantial^ly. ' 

In Figure 3.3. the information from the placement Service is 
presented in another way. The number^, of vacancies and applicants registered 
is presented by year and by degree of relevance to educational research 
(direct, facilitative, and not related),28 Particularly apparent is the 
crossover occurring from 1968 to 1969 in the "not related to RDDE" category; 
the number of applicants increased sharply while the number of vacancies 



28it should be noted, particularly for 1968. that the numbers of 
applicants and vacancies in Figures 3.3 and 3.4 do not sum exact y to 
the numbers in Figure 3.2; this is because Jes^nptjons of certa n 
vacancies and apj^licants were insufficient to cflassify tnem as directly 
related to. facilitative of, or not related to RDDE. 

\ 
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1968 * 1969 ^ ' > 1970 



■' 'Year 

• \ 

Fiqur^ 3.3 - Number of Vacancies (V) and Applicants (A) in the AERA 

\ Employment Service: By Year and by Relevance to Educational 

; ' Research and Research-related Activities (RDDE) 
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moved in the reverse direction. Also noteworthy are the pronounced 
reduction from ms to 1969 in applicants for positions directly rfelated 
to research, and the intrease'in applicants for^RDDt f z^ci 1 i tati ve posi- 
tions from 1969 to 1970. ' \ 

The information presented in Figure 3.3 seemed to reinforce the 
authors' opinion that the 1969 ^ta tended' to obscure supply and demand 
trends. This is due in part toV^eeling that the participants at the 
1969 ^inntJdl meeting 'may have been substantially different as a group than 
the more ."usual" AERA annual meeting population (assuming that the 1968 
, Chicago and the 1970 Minneapolis meetihgs more likely were attended by 
the "usu^" 'population). / A second reason is the dramatic reversal from 

fx 

1968 to 1969 in the numti'er of vacancies and" applicants for positions not 
related to RDDE. Additionally, the large reduction in applicants for 
positions directly related to research reinforces this concern ^ 

/For these reasons Figure 3.4 was lyepared to show the data from 
the/f968 and 1970 AERA employment services only.^^ In this figure it 
can be noted that for positions directly related to RDDE the ratio of 
applicants to vacancies has remained relatively cbnstant over the two-year 
period, although there' has been a reduction in the absolute number of each 
' The reductions in absolute numbers can be explained in part the reduced 
convention attendance in 1970 as compared with 1968. Also in Figure 3.4, 
the ratio of vacancies to applicants for positions facilitative of RDDE 
has remained nearly constant, with nunters of applicants and vacancies 



' 29it was assumed that (a) trends could be best identified if 
populations .'femain fairly stable and (b) supply-demand trends for one 
population (e.g., upper Midwest) would be similar to trends for another 
population (e.g., West Coas^). 
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Figure 3.4 - Number of Vacancies (V) and Applicants (A) in the AERA 

Employment Service: By Year and by Relevance to Educational 
Research and Research-related Activities (RDDE) 
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increasing slightly over the two-year period. Finally, for the positions 
not related to RDDE, it is apparent that vacancies h^ave reduced sharply 
while applicants have increased moderately. ^ * 

One final presentation was undertaken to highlight the areas 
of most concern to the AERA Task Force: the vacancies and applicants for 
1968 and 1970 were plotted after the two categories inore relevant to 
research (directly related to, and faci litati ve of RDDE) were combined. 

In Figure sls, the relative stability^of the applicant to vacancy ratio 

\ 

in research and research faci litati ve , Categories is j displayed. The 

I 

crossover between numbers of vacancies and applicants in the non-RDDE 
category is, of course, identical to that in Figure 3.4. 

Upon reviewing the data and figures in this section, it is diffi- 
cult to present definitive statements on supply and demand in RDDE, based 
on the 1968-1970 AERA employment service data. The limitations of the 
data enumerated in the opening remarks of this paper, plus the vagaries 
in the data introduced by changing annual meeting sites and the time of 
yaar for the meeting, make interpretation difficult. Note, too, the 

pressure on apprehensive job-seekers as they com||lete self-report forms 

I 

that they know prospective employers will read. \ 

General statements and conclusions will be^presented in abbreviated 
form inthe last section of this chapter. With this general overview 
of supply and demand in ml-Rdj consideration is now given to trends in the 
skills required by employers as compared with ski'lls "possessed" by 
applicants. ^ • o 
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Trends in Skills Required by Employers as Compared with 
Skills Possessed by Applicants 

In this section, the same data will be considered from two 
different perspectives. The first treatment of the data is relatively 
straightforward and consists of reporting the 1968 and 1970 percentages 
of applicants listing a particular competency and of employers requesting 
a given competency. The second treatment is less direct. In effect, tne 
same initial data are adjusted to reflect vacancies per applicant and 
applicants per vacancy in comparable units -for both 1968 and 1970. 
Result/. from the 1970 phone interviews are then compared to outcomes of 
the two treatments. The two resultant analyses, as one would expect, 
produce similar findings, yet these findings are expressed from quite 
different perspectives. ^ 

C'lan qes in Percentages of Empl oyer- requi red Compe t'en^es^ 
Indj^gl i cant-listed Competencies . \ 

In Table, 3.24 data are reported for 1968 and 1970 on the percentage 
of vacancies requiring specific areas of competence and the percentage 
of applicants listing specific areas of competence. For each year the 
percentage, add to 100 percent within each category. 30. (For example, the 
firs^ column represents the percentage of. vacancies that required each 
area of competence in 1968; the column adds to 100 percent.) 



30to perfonn this analysis, the last area f competence reported 
for 1968 and 1970 (namely, the category expressively dijbbed Other ) 
omitted, as the 1968 "Other" responses could not be categorized^ as rosoarch 
related, research facilitative or nonresearch-related. . ... 
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Several dramatic shifts f.rom'1968 to 1970 may be noted in the 
vacancies. Consider firstthose competencies listed in wnich a 2 percent 
or larger decrease occurred" in the "total" category (2 percent having 
been chosen arbitrarily as an indicator of marked change). 

^ decrease .from:' 



Re'searct^ methodolo^gy 7:0 to 2.4% 

Researchl design ' / • 8.4'to_4.3% 

Survey/insti-tutional research. ' 3.8 to '0.9% 

, Evaluation techniques (general) • 9.8 to 5.5% . 

Testing/Applied measurement 4.6 to 1.3% 

' Elementary statistical techniques ; 9.8 to 1 .5% 

Research idmi ni strati on/manageiDent . 4.4 to 2.3% 

Sunmirig over the slubcategories within research, evaluation, measurement, 

and statistical analysis makes the downward trends even more apparent. 

In 1968, for exampl^, 24.8 percent of the vslcancies required conipetency 

in research methodskypes , while this figure dropped to 12.3 p-^rcent of 

the vacancies in 1970. -The total trend over both evaluation categories 

was also markedly doin (from 12.0 percentin 1 968- to 6.1 percent in 1970), 

especially for positions directly related to research (16.6 pe'rcent in 1968, 

4.5' percent in 1970)>:1 The overall decrease in measurement from 13.6 percent 

to 7.9 percent was pa^icularly eyident'^in the research-related and 
■\ \ • - / I 

research-facilitative Categories. A dramatic reduction also occurred 

.]-.'■ ' 

in the percentage of vacancies rex^uiring competence in statistical analysis 
(11'.8. percent in 1958 to 2.1-i3ercent in 1970). It should be noted in 

• passing that- di f fusion '^s -an area of competence essentially was ignored 

1 31 
by employers or applicaihts over the two years examined. 



31as was noted tn Chapter 2, "perhaps- the proliferation of roles 
for diffusers embodied iln current literature on educational change is^^ 
prophetic rather than dejscriptive of present professional priorities. 
This conclusion was based on tne 1970 phone interview f employers' and 



Obviously, given the pr'onounced decreases above in tne "research'/, 
evaluation/statistics domain, there^had to be porresponding increases 
in other areas. ■ Tne types of cqmpetehcies for.w|iich there was a marked 
increase (-2 percent or more) in demand in 1970 vacancies over 1968' 



included: 



increase from: . 

— ^ — 

0.4 to 2A% 



Educational development . 

tduqati on/school psychp^ogy 2.8 • to 10.7;: 

educational ^ociolOg^f/economics _ 0'.8 to '2.8,^ 

. Curriculum develo^taent/an^lysis 1 .6 to 4.1. • , 

Teacher educatiort/inservice training 5.2 to 8.3,.. 

Subject matte|>^reas • 1-0 to 8.1. 

Special edu^i on ' ' ; , 0-4, to 5.6. 

InstructWt>a1 media/technology ^ _ 1.-1 to 5'.8-o ^ 

It. is noteworthy that many of these increases, came about via vacancies \ 
listed in ^t'lie categories of directly related .to/ and facilitative of 
researc/. Also note that several of the .increases involve development to"- 

a gre,4ter or lesser degree. - >- 

y • Turning the focus to competencies reportedly possessed by applicants 
to the AERa' employment sepvices , .total • decreases of 2 percent or more from 

1968 to 1970 were seen in:; , ' ; • ' / ' 

• . / • / ^ 

. decrease from 

Research metnodology , - - 12.8 ■ to . 1.7 ■ * 

/ . ' Curriculum development/analysis . 6.5 to 4.5;:.^. 
■ Teacher eclucation/inservice training 7.9 to 4.5'r 



31(contiT)ued) their view of diffusion, oh the average, as ce atiyely 
less important than pther functional areas of RODE Notmnghn the 
1968 or 1970 AERA employment service data refutes the position suggested 
by'th^s quote, unless rt is the incr^^ses in the areas of in|truct>ona 
media/ technology arid writi-ng/editing skills ,(in research-reUted areas). 



The number of competencies for which pronounced percentage decreases 
were observed is smaller for applicants than for vacancies. In addition, 
the shifts downward are less dramatic, with the exception of research 
methodology. Note also that only in the case of research methodolpgy is 
a pronounced decreasle in, vacancy competencies matched by a decrease in 
applicant cofppetencies, In^ the other two aireas at decreased applicant^ 
emphasis, there was increased demand .on the part of employers. 

Jhose areas in which an increased percentage of applicant! 
reported competence Include the fo'llowtr 

increase from : 

; Educational research " ^0 "to 10.^ 

Research design ' . 3.0 to 5.7% 

Evaluation techniques (general) 3.9 to 5.6% 

Subject matter areas 0.8 to 4.2% 

Instructional . media/ technology 1-1 to 3.1% 

Two of the five areas (subject matter areas and instructional 
media/ technology) agree with the changing demands represented by the 
vacancies listed. Hpwever, the other three competency areas (educational 
research, research design, and evaluation techniques) represent movement 
in directions bppbsitej to ^t+ie demand, trends . 

Although' some |)ortion of these percentage shifts is probably due 
to the changes |in the^personnel using the employment service over time, 
the very large shifts must have some additional explanation. In the 
sections which fdllow, the* results of this analysis wilTbe compared witn 
the 1970 telephoneXinterview survey to determine points of coimionality 
and disagreement. Consideration will now be given to the second analysis 
which adjusts the competency data in order to make comparisons, 
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Changes in the Number of Vacancies per Applicant from 
1968 to 1970 in Relation tot Area of Competence . ^ 

It is a straightforward matter to" use the vacancy and applicant 
data from the 196^ and 1970 employment services to c(xnpute ratios of . , 
vacancies to applicants and of applicants to vacancies for each area of 
com^tence. This has been done in Table 3. 25, 

Unfortunately, interpretation of the table is not nearly so 
'straightforward. Two basic problems are apparent"-' (First, tne tabiy'^is 
not divided by relevance'^tq RD\£; in fact it is^not possible to construct, • 
Table 3.25 to include^such a division, since 185 applicant forms and 
146 vacancy forms- for 1968 could not be coded on^tfiis dimension. 

A second problem might be stated asjack of comparability of 
units in Table 3.25; i.e., vacancy units and applicant units' are comparaole 
neither within years n^ across^years. Recall that each applicant "could 
list one or more areas of competence on his self-description form and tnat- 
an employer could list one or more competencies required for each vacancy. 
Given this -option, tKe number of co,mpetencies indicated varied considerably 
and the totals from Table 3.25 are far greater than the actual number 
of vacancies or applicants. In Figure 3.6, the average number of compe- 
tencies listed by applicants and employers' Is plotted for 1968 and 1970 

32 

and for the three categories indicating degree of relationship to RDOE. 

In Figure 3.6 the pattern is clear. From 1968 to 1970 and in 
each category, the, average number of competencies required for vacancies 
decreased while the average number of competencies reportedly possessed 



32comparable data for 1969 are available but are not presented for 

the reasons given prev"^ously, i.e., the assumed unrepresentative nature of 

that annual meeting and--more important— the difference in. the employmeht 
service forms for that year. 



in 

CVJ 



CO 
01 



to 



CO 

O - 



u 



o 

u 

-M C 
U (U 

Q. O 



a> 

o 

c 

c 

>- 

<: to 



o 

4-> 



>» > 

rO . 00 
U 

to -M 

>» c 



\ 



to 






1" 


c 






a. 


u 


e 










CX 






a: 


<: 


UJ 




< 


-o 




c 




fxs 








to 




(U 


1. 


•r- 




O 




c 




to 




u 
















o 
















E - 













u 
> 

CO 



< 



5 



u 
*> 

CO 

c 



00 



1 




Ln 


en 






• 




• 


• 




o 


o 


' o 





\ 



o 



CM 



O 



in 



VO 00' 00 CM CO* 



csj a\ 



CO 



in CO CO. l6 CO 



O, lO O 
OJ O CO 



in rN. CO 

O r— O 



O CO O 

o 



CSJ 00 CVJ 

^ 



^ IP 
CO o bj 



CTl O 



in 



x: 
o 

(D 

to 

(U 









>^ 






4-> 














to 


C 






(U 




o 




sz 




£= 


U 










o 




O 


c 




Q> 


'o 






a 




(U 




in 


T3 


c: 




o 


to 


-M 




Q> 


O 




3 




3 


(U 


to 


or 












Q. 


T3 






to 




> 




om 


ho 




ne 




3 L 




5 


o 




c 






C 










o 




x: 


»— i 








1? 






a 


\ 


'to 




O 










>> 


c 


C 








to 


to 




o 


a 


to 


u 


U 


<u 


<D 


> 












to 


to 


















3 






<: 


Rese 


UJ 




m 


Educ 


Educ 



4-y 

c >, 

to u 

u c 

u 

Q. »t5 
CX?> 

s- 

in 
to 4^ 

(U c: 

•I— lO 

u u 
c: *i- 

tO f— 
U Q. 
CX 

> <c 

O O 

J- s- 
(U QJ 

■i-i 

3 3 



to 



c > 
o 

* CX to 
c 



to CL 

(U a. 

'I- to 
u 

c o 

<o c 
u 

^ to <D 

> $- 

^ % 

o 

(D (D S- 

JD x: o 

E 4J CD 

3 (D 

>> 10 «/) 3 <J 
U O (DO 
C T- 4-> JC 4-> 

U Q. U £Z 
to CX C 

C > C 



T3 
Ci 
3 



4J 
C 

o 
o 



ERLC 

iifiiinniiiTirfTiTiaiia 



192 



r 



<u 
u 

> 

CO" 

c 

0) 
E 

E 



O 



U 
> 



O 
E 



00 



c 

a. 
E 
o 
o 



to 



CO 

o 



C\J 

CO 



cn 

CVJ 



* 

CM 



\ o 



<4- 



o 



00 



CO 











CVJ 










o 


o 


o 


















* 


CO 




CNJ 








o 


ro 


CO 


o 


CO 




r— * 












CVJ 




# CO 


CO ' 






• 










\ 








o 


00 


o 


















CM 


o 














o 


O 



C) o 



o o in 
* • , • 

00 CO <Ni 



^ CM VO 



00 CO 

/ 



o> o> 

O CO 



GO 

o 



<M 
CM 



> 

UJ 



o 



CVJ 



U3 


OO 




00 




1^ 






o 


IT) 




CO 




o 




CVJ 






CO 














i 
















































c 




15 






c 














c: 




















o 




















*^ 






Oi 


c 








*o 










s- 


o 








s- 




u 






3 










cr 


CU 


3 






to 






u 




o 












u 






ns 












(U 


3 






U 


o. 




to 






s: 




\A 


to 


•r" 




c 






o 






•f* 




-M 


to 


JZ 








-o 


to 






to 


<u 


u 










c 




fO 


•r- 


3 


<u 










o 






-M 


cr 




O 




o 




o 


'IS 


00 
























C 


c 


c 








c 


< 


00 


CO to 


-C 


o 


^ o 








C <D 




S- (U 




- u 








ii 




(U E 




<0 3 


-O 3 






t5 4-) 


c 






E OL 






<u cr 






C ra u 


(U 


s- o 


C 


3 O 


u 




u 




3 


<U 3 3 




3 ^ 










,c c 


i« 




C r-' -O 




to u 












(U 




(U ra o 




<o >> 


to 


to > 


to 


(U u 


. > u 


-M 


> 


O) > s- 


3 




<D 




*r— 




-o <u 


3 


LLJ 




to 








-M 




< H 


Ol 




















E 






1! 








St 






Co 



/ 



193 



U 
> 

00 



c: 
E 



O 



=«fc 



T5 



O 

o 



to 

CM 



o 

> 
CO 

c 

B 

>i 

O 



CO 









* 


















- 


• 
















f 


























































co- 


IT) 




CM 


00 


CO 


CM 




in 










• 














o 


O 


o 


O 


CD 


o 

♦ 


o 


O* 


o 


o 










• 








* 




o 


O 




crj 


CM 


o 






o 


























CM 


CM 


IT) 




,co 






CM . 








as 


CM 




CO 














CO 








1^ 




CM 


o 


r ■ 














- 


* 














- 


% 

CO 










in 






































(3 






•to 


o 






o 


o 


























o 

r 


o 


o 




CO 


o 






o 
















• 






o 






CO 


CO 


CO 


CM 




CO 
























CM 


"cm 






o 


o 


cr> 


P 


O 












































o 


o 


C30 




CO 


tn 


CM 




CM 


CO 






lO 












CM 
























1 










e 












CM 


* CM 


00 


CM 


00 


o 


CM 


lO 




CNJ 





















c 
o 
o 



c 



t/) 
C 

lA 

E 
a> 
4^ 

to 

00 



a 
o 



CO 















SDLUl 
















o 
















c 






o 










o 
















u 






'o 


ro 








Ui 






_c: 








C 
















•r- 


>? 




o 








o 








o 


lA 


E 






'oJ 


o 




x: 


Q- 




o 




■ v> 
















c 


'o 




00 




a> 


'o 


a 










ro 




jc 


>» 


o 


u 








K 


a 


(A 




o 


c 




C 






D. 


■o 


00 


o 


o o 


i 


o> o 






c 








C3. 




rO» 


fO 


ro 


ro 


^ o 


O 


•r- O 


<— • 


a 




c 




ro ^ 










a> 


o 




a a 




u u 




c 


a 




lA 


3 >> 


> 


ro >^ 


a 




c 






"O to 


8 


<U iA 


o 




fO 


10 


* c 


Ui (X 


^ £1- 


00 

2 


o 


Guid 


Educ 


Admi 



c 
o 

ro 



-si 

rO 

<A C 
0) <0 



c 
* o 

ro 

4A 



E 



ro 

c 

CD 



ERLC 



194 



a; 
u 

> 

a; 



E 
LU 



(U 
C 



o 



CM 

s 



'5 



0) 

u 
> 

(U 
CO 



, o 
o 



CO 



0) 

u 
c 

(U 



a;, 



CO 

o 



CO 



CvJ 
CM 



ro 
o 



00 
CO 



o 



c 
< 

c 



o 
> 



O 



o 



in 



O 



CO 



o 



o 



CO ID 



O , r- 
CO CO 



f-^ o 



o 



f— . CO 



o 



CO 
CSJ 



f— o 



*^ 00 CO tr> U3 

CVJ f— 



CO 



o 



XL 
U 

01 



U 



C 

o 

U 
O 



in 

(U 



3 



00 ^-r 



u 
o 

00 



C71 

4 

(U 



00 



7 



o 

c: 





u 




> 


5 




(U 


C 




























to 




s. 
















T3 


LU 






o 


(U 








51 




o 










u 


o 












U 


to 








rs 


c 








■o 


o 






LU 




< 




sz 








<D 


Cl i 




u 


CJ 




o 


fO 




c 




</) 








s» 




U 








« 


<D 






x: 




CX 


c: 




4J 








3 


O 





ID 
O 



00 



00 



00 
CO 



00 
O 



CO 



00 



O (U 

O 
U (U 
3 CX 

<U -I- 



5.00 



4.50 



4.00 



U 

c 



^ 3-50 



a; 
a. 



o 



a; 

E 
2J 



> 
< 



3.00 



2.50 



2.00 I 



1.50 



1.00 ^1 



t 




Directly Related to RDDE 
Facilitative of RDDE 
Not Related to RDDE 




1968 



1970 



Year 



s 



Figure 3.6 - Average Number of Competencies Listed by Applicants (A) 
.and for Vacancies (V) in the AERA Employment Service:^ 
B.y. Year and by Relevance to Educational' Research and 
Research-related Activities (RDDE) 
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by applicants increased. One interpretation of the reduction in the 
'averl&ge number of competencies required by employers is that it may 
reflect a certainty about the type of persen being sought. That is, the 
employer might be hiring to fill a specific opening in his organization 
Tposs»1b1y to fill an established position whoseMncuinoent Is preparing 
'to yacate); the employer cannot afford the luxury of hiring a jacT<-of- 
all-trades. Conversely/so this interpretation goes, in expansive jti^nes 
when many staff are being sought by the same organization, it is possible 
to hire persons less restrictively and then redefine the remaining positions 
in terms of those area^ of competertce not yet "covered." 

-The increase in the average number of competencies J isted bV the 
applicants, oh the other hand, might be explained by the gradual imppve- 
ment in training in these areas of competence. A second interpretation is 
that the applicants are influenced by the prevailing dialogue proclaiming 
a tight job market. To avoid upemploymentythey list more areas of coippe- 
tence/f and this both improves their overall appearance on paper and 

increases the probability that they will be considered for more joDS ^han 

' / 

if they had listed only one specific area of competence. ^ / 

Be this as it may, the problem still remains of presenting the 

"area of competence" data from Table 3.25 in comparable units Doth across 

years and between the vacancy and applicant breakdowns for a single year. 

in 1970, for example, 726 applicants listed V,697 co^npetencies while the 

412 vacancies ijivolved only 569 competencies. In, orl^er to avoid the 

-distortion which would result from use of the raw tiafa, it was necessary 

to devise a common unit for presentation of the vacancy and applicant data- 

The procedure followed was to multiply the number of apparent vacancies 

(or applicants) within a given year and w/thin a given category of 

/ * 



research relevance by an adjustment** percentage • That ijercentage. was 

determined by dividing the number of actual vacancies (or applicants) 

in that category by thre total number of competencies in the. same category* 

y ' % 

This had the effect of reducing the number of competencies within a 
ategory to equfvalency with the number of*actual vacancies (or applicants) 
in that category. Cle^Ty 4;his procedure does some violence to the - 
data> yet it is felt that general ^omparisotis can be made with the data 
thus adjusted. 

In Table 3.Z6 adjusted numbers of vacancies and a|ipli cants are 
presented by ^ear and by research-relevant category* -Table 3:27 follows 
directly from Table 3.26 and in it i's indicated the adjusted number of 
vacancies per apaVica/it (also adjusted) and^b^ adjusted number of 
applicants per vacShcy (also adjusted) foi[;J%8 ind 1970 by re'^sfearch-^ 
relevant category. - Wis apparent In the tables that the various numerical 
transformations resulted In some distortion of 'the data. Neve rjthel ess, 
tte entries iff the two tables are in comparable' uflits. 

From Table 3,27 it is pos^sibje to classify the changes in vacancies 
per applicant from" 1968 ^ 1970 into three categories.: (1) pronounced 
reduction in vacancies per applicant; (2) little change in vacancies per 

applicant, and (3) pronoiinced Increa'se In vacancie§ per applicant. The 

' i 

pronounced increases and decreases are shown In Table 3*28. If vacancies 
per applicant either increased or decreased by a^ctpr of 3 from 1968 to 
1970, then an entry was^made in Table 3. 28, Also incli^jled in^the table 
are ch/nges of 1.5 vacancies or more when the compari^n figure f rcNn tne*^ 
other year was zero. - The reatler is cautioned that reference#to other 
tables' pften is necessary to keep the entri^es in Table 3. 28, in perspective. 
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for the latter are ratios and must be interpreted in M^t of the 
absolute and/or adjusted number of vacancies and applicants from 
which they are derived. 

It can be noted in Table 3,28 that the pronounced decreases 
clearly outnumber the pronounced increases; additionally, the decrease 
ratios are larger than are the increase ratios- Examination of the 
areas in which pronounced decreases occur reveals that most of the 
categories concern what might be called the research-development- 
evaluation complex rather dir^ctl>* The decreases often cut across the 

I 

three categories within the area of competence; that is, the decrease is 
general across directly research related^ research facilitati ve, and 
nonresearch related activities. 

Dramatic shifts downward occurred in mafny areas of competence 
such as educational research, research design, survey research, evaluation 
of instructional products, instrument construction and development, 
elementary and advanced statistical techniques, systems analysis, etc*; 
note that reductions in these^^reas often occurred in vacancies directly 
related to, or facilitative of research. Reductions in most of tne other 
categories (compter techniques and programming, learningjfod experimental 
psychology, social psychology, clinical psychology, educational sociology/ 
economics, general administration, curriculum development and analysis,, 
teacher education/inservice training, subject matter areas, instructional 
media/ technology, and writing ability/editing) occurred in vacancies not 
related 'to research. Conversely, many of the pronounced increases dealt 
with these same areas of competence but with the directly related to, 
and facilitative of research categories: Note too (in Table 3.26, the 
directly related to, and facilitative of research columns) that the 



Table 3,28 

Pronounced Changes in Number of Vacancies per Applicant from 1968 to 1970 
By Area of Competence and Degree of Change 





1968 


1970 






A# 


A# 


Research-Relevance 


\ Area of Competence 

\ 


VA 


VA 


Category 


Pronounced Decreases 








1 Educational Research 


2.9 


0.2 


DR 




1.0 




■ Total 


Research Methodology 


1.3 


0.4 \ 


F 


Research Design 


1.8 


0.5 ^ 


V DR 






0.3 


^ F 




1.6 


0.4 


Total 


Survey/ 1 n^s ti tuti ona 1 


37.0 


8.3 ' 


DR 


Research 


1.6 


0 


^ F 


./ Evaluation Techniques 


40.0 


4.1 


Total 


1.3 


0.4 


DR 


/ 

/ 

\ 


2.2 . 


0.7 


F 




1.5 


0.5 


Total 


Evaluation of Instruc- 


7.6 


0 


DR 


tional Products 


10.0 


0.7 


Total 


Testing/ Applied 


1.0 


0.3 


F 



•Measurement 



A#VA: Adjusted Number 6f Vacancies per Applicant. 

DR: Directly Related ko RDDE. 
F: Facilitative of RODE. 

NR: Not Related to RDDE. 
fotaT: All Categories 
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Table 3.28 (Continued) 





lyoo 


1 Q7n 






nW 


Mtt 


i\cbt:arcil~i\ci cVaiiCc 


Area of Competence 


VA 


VA 


Category 


Instrument Construction/ 


11.0 


0.9 


V DR 


Development 


2.0 


0 


F 


• 


' 1.7 


0 


4 




19.0 


0.6 


Total 


Elementary Statistical 


0.5 


0.1 


'dr 


Techniques 


0.4 


0.1 


F 




16.0 


0 


NR 


1.0 


0.1 


Total 


Advanced Statistical 


4.2 


0 


DR 


, Techniques 


3.0 


0.5 


• F 




1.7 


0 


NR 




4.4 


0.2 


Total 


Computer Technology/ 


30.0 


0.8 


"nr 


, Programming 






- 


Systems Analysis 


4.5 


0 


DR 




1.7 


0 


NR . 








1 ULd 1 


Learni ng'j^Experi mental 


13.0 


0.3 


NR 


Psychology 








Social Psychology 


^ 5.1 


0.4 


• NR 




4.5 


1.2 


Total - 


Clinical Psychology 


15.0 


0.6 


NR 


• 


r 


Q.6 


Tbta^ 


Educational Sociology/ 


7./ 


1.0 


NR 


Economics 


5.4 


1.5 


Total 








/ 
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Table 3.28 (Continued) 



t 


1968 


1970 




Area of Competence 


A# 

VA 

it 


A# 
VA 


Research-Relevance 
Category 


General Administration 


2.2, 


0.2 


NR 




0.8 


0.2 


Total 


Curricdlum Development/ 
Anajysis 

/ 


.0.8 


0.2 


NR ^ 


Teacher Education/ 
Inservice Training 


5.9 


0.7 


NR 


Sjjbject Matter Areas 


O 1 

o. 1 


1 .1) 


KID 

NR 


Instrument ftedia/ 
TechncMoiiv * 


5.9 


1.0 


NR 


/ Writing Ability/Editing 


3.2 


0 




Pronounced^ Increases 








Research Methodology 


0.3 


2,0 


DR 


Educational Development 


0.2 


0.7 


DR 




0 1 


0.7 


F 


f * 


0.2 


0.7 • 


Total 


Evaluation of Instructional 
Products 


0 


1.7 


NR 

• 


Computer Technology/ 
Programming 


0.3 


1.1 


F 


Educational /School 
Psychology 
> 


0.1 
<0.1 


1.0 
0.6 


DR 
F 


* Developmental Psychology 


0.2 


0.8 


.DR 


Social Psychology 


1.2 


4.4 

i 


DR 
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Table 3.28 (Continued) 



Area of Competence 



1968 1970 



A# 
VA 



A# 
VA 



Research-Relevance 
Catetory 



Educational Sociology/ 


0.8 


2.4 


DR 




Economics ^ 


0,3 * 


1.7 


F 




General' Administration 




0.3 . 


., DR 




Curriculum Development/ 


' .^0.1 


0,9 


/ . ' DR 


\ 


Analysis 










Teacher Education/ \ 


, <0.1 


1.0 


DR, 




Inservice Training \ 


° <0.1 


1.4 


F 




Special Education 


\ 0.2 




DR 






^, 0.5 


1.5 


Total 

/ 


Instructional Media/ ■ 




0.7 - 


OR 




Technology 




1.8 


F 




Writing Ability/Editing 




8.3 


DR 





\ 



ERIC 



pronounced increases often involve areas of competence where tfje 
absolute numbers of vacancies and applicants are relatively small. Thus, * 
the large reduction of vacancies in nonresearch-related positions noted 
earlier in this paper is concentrated in many of the areas of competence 
in which, at the same time, there has been a small but noticeable 
increase in the number of vacancies in research and research facilitative 
categories. ^ 

Comparison of the Res4ilts of the 1970 Telephone Interview of - 
Employers with Analyses of the 1968-70 AERA Employment 
Services Data . 

" ' ■ * 

I 

It is of interest to note correspondences between the two analyses 
above (ff the 1968-1970 AERA employment services data and the results from., 
the 1970 telephone interview of employers. Neitf^r of the investigations 
{the AERA employment servic.e s^udy and the telephone survey of employers) 
was designed to Verify data from the other, since different populatipns 
were sampled, different questions were asked, etc* Thlis there i$ no^ 
reason to expect high correspondence between the sets of information ^ 
generated* At the same time, extensive disagreement would not be expected. 

The 58 employers who were interviewed by telephone identified 
skills that they considered important in their organization and, of the 
> important skills, tlenoted those that they felt were in short supply or 
"hard to come by,** In some cases, the skills judged in short supply-by 
the employers bore a high degree of correspondence to the areas of compe- 
tence generated from the' placement data. Ten of these skills are listed 
below: 



1) Understanding and using ^perimental design and other 
approaches to inquiry (reseajjchjj . --^ ' 



2 
3 

4 

5 
6 
7 
8 
9 
10 



Developing measurement instruments (research). 

Reporting research findings and implications orally and 
in writing (research). 

Conceptualizing* system^ their elements, and interrela- 
tions among these elements (research-based development). 

Selecting or devising appropriate techniques for measur- 
ing outcomes '(research-based development). 

Composing infonpation for accurate and pervasive dissem- 
inati6n (diffusion) , ^ . 

Devising and conducting long-range evaluation of the 
i/istalled package (diffusion)- 



Measuring current actual outconje? of ♦the system 
C^ontext evaluation). , ' . '^^ 

*■ 

Applying appropriate designs tp evaluation studies - 
(outcome evaluation)/ ' - ■ 

Selecting (or developing) and using techniques of measure- 
ment to yield information relevant to standards (outcome 
evaluation). 



For the purpose of comparison, it was assumed that these skills 
shared common characteristics with certain areas of competence^ as indicated 
below. 



Skill 



1) Understanding and using 
exDferimental design. 



2) 



Developing measuring 
.instruments. 



3) /Importing findings, 

orally and in writing. 

6) Composing information 
for pervasive dissem- 
ination. 

4) / Conceptualizing systems 
*v,and interrelations. 



Area of Ccmipetence 
Ic) Research design ♦ 



5c) Instrument construction 
and development - 



19) Writing/editing 



8) Systems analysis • 



^ 213 



5} Df vising appropriate 
measurement techniques 
to measure outcomes* 

8) Measuring current actual 5b) Testing/applied 
^outcomes. measurement. 

10} Selecting measurement 
techniques to gather 
data relevant to' 
standards. 



71 Designing and g0riducting 
long range evaluation. 

9} Applying appropriate 
designs to evalujftion, 
studies. 



Evaluation tecnniques. 



' Focusing on the relationship between the telephone Interviews 
and th^ first aoalys is above (changes in percehtajges of em^loyer-^requined 
competencies and applicant-listed ccHnpetencies) , attention should be 
redirecte^d to Table* 3-24. Research design, instrument construction and, 
development, testing/applied measurement, and evaluation techniques clearly 
received less eoiphasis in the 1970 vacancies than in those' listed for 
1968* For systems analysis, the employment service data was mixed across 
the research -re/1 ated, research facilitati ve, and nonresearch-related 
categories, with an overall trend of slightly downward. Only in the case 
* of writing/editing do the data presented, in Table 3.24 suggest that the AERA 
employment service vacancies reflect the telephone interviewees^' perceptions^ 

of an important skfll in short supply- 

. / ... 

Turning to the second analysis and the areas of compfetence in 
Tai)le 3.28, it is seen that each competence area under discussidh experienced 
- a, pronounced change from-1968 to 1970. However,* in every case except one 
the change was a decrease in vacancies per applicant. The one skill which . 
showed an increase for research-related vacancies (i^.e., writing/editing) 
also -showed a decrease 'for the nonresearch-related category. 



( 



Thus, while the't^o analyses of the AEFtA employnient service* data 



agree very well with one another (as one would expectj, theri? is little 

agreement between t\ie^ an^d the results of the telephone interview survey. 

On. the one hand, the telephone interviews indicate that the 10 skills 

' listed above are important and that they are in sHort supply. ^ Thus it \. 

• might 'be inferred that vacancies exist which require the exercise of t/es^fe 

- ' . • ^' I : 

skills, The>4nployfrient service data, on the other hand, 'show that 
** ' ' » • 

percentage of vacancies refluiring these same 10 ski'lls has dejcreased > ^ 

since 1968 and that the Jiumber'of vacancies per applicant in these skill 

areas is also decreasing. , ^\ 

■ " It is' not easy to resolve the apparent discrepancy. Firsts tne 

two'sets of data were obtained from different populations, and aUhough- 

some overlap is likely, the AERA employment sample could well be metre 

d9minated by university personnel -than was the sample for tne telephone 

interviews. Second, the question "Which skills are both important and in 

short supply^" is fundamentally different from the question "Is your 

organization at this point in time attempting to hire persQrts-p^ossessing 

J these important, short-supply skills?" That is, even though employers 

recognized that a skill was important to their operation and was Jn short 

supply, they still, may not have been in a position to a^ a new staff 

member in that area even if one was available. A third 'gossibility is / ** 

that the telephc ^,e-.inter'viewed employers may have been sifitting much 

higher standards for, "competency" in an area than were thte AERA employment 

service applicants reporting competency in that same ^rea. , 

♦ % 



No attempt is made in. this final section* to review all of the 
foregoing commentary; rather, attention is focused on several salient 
points 'which seem warrar sd^^n spite of the great number of uncertain- * 
ties associated with the data, 
^ ^ . THere is a trend over the past two years tdWard fewer vacan- ' 

• eies bein$ re'gistered at the annual AERA employment service. The reduc- 
tionMa slight in the case of positions. related- to and facilltative of 
' RDOE (from 322 to 288), while the reduction is pronounced for vacancies 
.not related to RDDE (from 339 to 124).. - ' - ' . 

2s Jhere is a^trenfl over the past two years toward slightly ^ 
fewei^- appTitants registering at^the annual AEfVi employment service. Tne 
\ ifeducti on Mn^ applicants for research-related and iresearch-facilitati^e 
posjtions'is considerable (from 655 to 51€0 while the number of applicants 
for positions not related to RDDE-has' incre^ed '(fifom T21 tp 2^1)^. It^ 
is upclea^^ether this is- indicative of a change in the typ§s of personnel 
being trained, a change in the types of persons using tne employment 

\ . . , \ 

service, a change in the self-report tendencies of 'applifcants, or some 

> * ' "7 ■ 

combination of these or other factors. 

♦ « . '4 

3. Based on^ the number of vacancies listed in the 1970 A^RA 
employment service, there is a general oversupply of- applicants in most , 
categories/ "The ratio of applicants to vacanciies ^for positions directly 
related 10 and .facilitative of R^peThas changed slightly in the direction * 
of ffewer <ippli cants per vacancy (from 2.0^ applicants per vacancy in 1965 
,to 1*79 applicants* pe-r vacancy in 1970). Despit|;^is small improvement, 
' the.AERA ismploymefit servic^ data seem to contain 'eaVly indi^cators of wnat 
co^ld"develop into a depressed job market situation. For example, in 
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Figures 3.3 and 3,4.» it can be noted that applicants outnurnb^r -vacancies 

in three categorils (directly related to,- facilitative of, jand 

not related to RDDE)- Additionally* fewer transactions are- ocscurring 

on the job market; the absoli^e numbers fif both vacancies and/applicants 

are down. In comparing th6 1968 and 1970 data, this is particularly 

true for vacancies (769 to 412) and Tess pronounced for applicants 

(811 to 727). ^ • 

,4. It would appear that the current overs upply of appl4cant5 Is 

especially pronounced in the following areas:. 

Educational Research »• 
Research Design 
Evaluation Techniques 
Evaluation of Instructional Products 
Testing/Applied Measurement 
Instrument Construction/Developinent 
Elementary and Advanced Statistical Techniques 
f Systems Analysis 

Guidance and Counseling 
General Administration 

Th% statements ^above (concerning areas of competence where a pronounced 

oversupply of applicants in/ relation to vacancies exists) must be tempered 

in light of the 1970 telephone interview study in which several of the 

* 

competence areas listed above wer^ felt by employers to represent skills 

both important and in short supply. 

On the other Itand, tK^ose areas in which there is an und^r supply 

of applicants are not as numerous. Areas of competence in which there" 

is an undersupply* or a relatively good baladce between applicants and 

vacancies are: . ^ 

Survey and Institutional Research ■< 
, Social Psychology 
Educational Psychology 
. , Vocational Education 
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Subject Matter Areas 
Special Education 
<^ Instructional Media/Techniques ' . 

Writing Ability/Editing 

^5. Although there currei^tly exists in the educational literature 

d good deal of emphasis on accountability j perfo^nce contracting, the 

necessity of both forrnative>and summative evaluation in educational 

settings, etc., and despite the pronouncements of the 58 employers taking 

part iji^he t-elephone interview, thfere is. nothing in the AEIRA employment 

service data to indicate that e valuators are irt greater demand in 1970 

than they were in 1968. In fact, the data from the employment servite 

suggest that tftey are in less demand. One -possible explanation is that 



school districts and other organizations, because of current financial 
^- . f ^ ^* 

restrictions, may have merely labeled*' some of their longstanding employees 



as "evaluators" rather than hiring new personnel recently trained in 
evaluation, - 1 

&• Although aftplicants for positions ^i^n development continue to 
outnumber vacancies {^ee Tabl^ 3,2Z^there*are some bits of information 
which suggest that development is emerging as a more important and larger 
occupational^ area than was pjreyiousll^ true. In the discussion*of t^ble 3.24, 
for example, it was noted th'at there was a modes't increase in demand in 
1970 for f)ersons having competencies in educational development » curriculum 
development/analysis, and instructional media/techniques. It would seem 
that confirmation of this trend is requii^d from other sources before it 
ran .be accepted as a definite indication of a growing demand for developers. 

7, Ebiployers interviewed by telephone indicated a need foy persons, 
trained, in di>f f us ipn^ skills. Yet these same employers ranked diffusion 
as less important than the other six RODE functional /areas (evaluation 
was presented in that interview in four siibareas rather than one). In 
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• ♦ 

addition, wh^ii checking the AERA employment service data for the last 

4 

three annual meetings, one- finds almost no mention of a demand for 
diff users or of a supply of4ppli cants in the diffusion area. It is 
clear-that educational^ literature" has been inundated lithi diff user titles 
such as "change agent." "facilitator," etc. Nonetheless, the only, 
support that might be garnered from the employment service data for the 
notion that diffusion is the coming wave, would' be the modest increases 
in the demand for skills "in instructional media/techniqUes and in 

/ . . " ' ( 

writing/editing skills in research -related areas ♦ 

;' 

8. In the discussion of Table 3.24, several shifts from 1968 
to 1970 were noted in the types of competencies , required by employers 
and the types of competencies reportedly possessed by /applicants • 
Although this time period is far too short to be certaiti, it would appear 

that the detnand (or vacancy) competencies are much more variable tnan tne ' 

/ i ^ 

supply (or applicant) eompetericfes. Jhis is as mig}\t be expected^ ^si nee 

the competencies required for a position can change almost overnight as a 

result of changes in funding, whereas manipulation of applicant competen- 

r J 

I 

cies is not so easy to accomplish since it involves, changing either 
direction or emphasfs in existing training programs, 

9. This final point should receive emphasis above all the others. 

! / 
The need for additional data on supply and demand-i-specifically data 

from sources other than the AERA employment service— is clear. • The 

vagaries attendant on a placement service that lacks a permanent site, a 

pemanent clientele, and a permanent staff are clearly pronounced, and 

data from such a source are difficult to interpret. Additionally, the 

AERA placement service is clearly dominated by applicants whj/ are university 

oriented ,in terms of the occupation they seek; liniversities also represent 



219 



the largest single employer group operating through the AERA placenient 
service. The lack of correspondence between the results of the telephone 
interview survey and the results of the employment service analyses may 
reflect in part the fact that universities were proportionally a smaller 
part of the telephone interview sample than they were of the employment 
service sample. . Other indications from the data also suggest the need 
for comprehensive collection of information f r<»ii /nany sources in order 
to establish reliable indications of supply and demand in educational 
RDK. 
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^ CHAPTEI\ 4 



AN ;»J^ALYSIS OF CHARACTERISTICS OF 1969-70 TRAINEES IN TITLE IV 
GRADUATE RESEARCH TRAINING PROGRAMS AND A COMPARISON 
WITH SIEBER'S STUDY OF 1966-67 TRAINEES 
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AiN ANALYSIS OF CHARACTERISTICS OF 1969-70 TRAINEES IN TITLE IV GRADUATE 
RESEARCH TRAINING PROGRAMS AND A COMPARISON WITH 
SIEBER'S STUDY OF 1966-67 TRAINEES 

Since the advent of the Elementary and Secondary Education Act 
(ESEA), the Graduate Research Training Program funded under Title IV of 
that Act has been the major vehicle for funding the training of educational 
researchers. Recently, the program h*as been under attack from many quarters. 
The U.S. Senate and the former U.S. Bureau of the Budget have both taken 
a highly critical posture toward -^the4)rog ram, and other internal reviews 
^ithin the Office of Education and the Department of Health, Education, 
apd Welfare have been quoted as being critical of the Graduate Research 
Training Program. In addition, several earlier studies (e.g., Clark & 
Hopkins, 1969; OiLorenzo, 1967; and Sieber, 1968) have been critical of 
either the basic structure and conduct of the program, the relevance of 
role types being prepared in it, or the background and quality of program 
trainees and directors.' 

This is not an attempt to answer or comment on the various criticisms 
of the program. First, It is an effort to furnish objective background 
information for both adherents and critics of the program.* And secondly 
it is an initial step toward determining the extent to which Title IV 
prognams provide training in the skills' and knowledge essential for edu- 
catlonal research. 

The focus of this chapter will be on cne major question: What 
are current trainees in the program like? The quality of outpJt from any 
training program is obviously dependent on two factors: the quality of 
the input (trainees) Into the program and th-e efl^ctiveness of the program 
In increasing their knowledge and shaping their behavior in 'desired ways* 
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The qualit^of the input is the prior question, since even an effective 

program will likely be effective in direct proportion to the quality of 

t(ie raw material entering the program, (While the foregoing statement 

mayN^ot hold for programs designed specifically for the disadvantaged 

or slow learner, it is felt that no Title IV progran^ were designed with 

such an intent.) 

In addition to describing current trainees in the program, two 

ancillary purposes remain. One is to compare current trainees with the 

* * 
1966-67 trainees (described by Sieber, 1968) to see if there have been 

changes in trainee characteristics in three years of program operation. 

A second purpose Is to compare trainees in "teminated" programs 

programs discontinued by USOE as of the end of the current fiscal year 

~ with the trainees in continuing programs* The comparison was conducted 

to def^rmlne the extent to which USOE decisions resulted in an overall 

"increase oip decrease in the quality of trainees in the program* 

It is anticipated that data presented ^ere will prove useful to 

several audiences, among them the following: USOE officials, where it 

might be useful input for major planning decisions; directors of research 

training programs; and AERA officials, especially itmbers of the Task 

Force and others with direct concern or responsibility for the training 

of research and research- related personnel. 

^he remainder of this chapter Is divided into five sections: 

(1) description of procedures used in collecting and analyzing the 

datat (2) description of some characteristics of the Title IV Graduate 

Research Training Programs, (3) presentation of prli^iary data on 

trainee quality and potential productivity, (4) presentation of other 

descriptors of trainees, and (5) concluding observations. 
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Procedures 

There were IJwo major sources of data for this study: (1) "Statement 
of Appointment of Trainee Under* the Educational Research Training Program" 
forms required by the USOE for each trainee, and (2) a supplemental form 
developed by the 'Vask Force staff and sent to directors of all Title IV 
training programs. , 

In Sieber's (1968) study of 1966-67 Title IV trainees, appointment 
„forfis served as the basic source of data. Specifically, Office of 
Education Form 6003 (2/66) was used. Sieber found these forms inadequate 
and in his study recomniended changes that would provide more data bn sub- 
sequent trainees. Most of his suggestions were incorporated into a 
revised Form 6003 (8-69), a copy of which is shown in Appendix In 
the present study, the. new forms were available for almost 90 percent of 
the 1969-70 trainees; the remaining trainees had filled out the older, 
obsolete .forms. Most data were comparable on the t^h forms, but where the 
new form provided data whicft did not, appear on the old form, the data was 
coded as missing for the 10 percent who had used the old form. 

A copy of the supplemental form developed by Task Force staff 
is also shown' in Appendix K. ' It was sent to proqra| 
directors to obtain information on the academic ability of trainees 



" These forms provided such information as the following: bio- 

graphical data, including birthdate, birthplace, marital status and nurrber 
pf dependents; educational background, including. institutions attended in 
the past and degrees received; employment background over the past five 
years, including positions held and/or duties, name of the emp.loyer and 
dates of employment; current or expected professional affiliation of the 
trainee following completion of the program; type'^of degree sought under 
the training program, when the trainee expects to receive that degree, 
and the discipline and subdiscipline of that degree; amdunt and type of 
financial support' received by the trainee; and a record of all major 
pii)li cations produced by the trainee. 
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(e.g., GRE or MAT scores and cumulative GPA) and the number of nonstiperid 

trainees in Title IV programs (trainees affiliated with Title IV prog^ms 

but supported by other funds, such as NSF or ND&A fellowships). 

« 

Collection of Data ^ 

In April, 1970 appointment forms for the 1969-70 trainees were " 
obtained through CQOperation ^f the staff of the USOE Research Training 

Branch. As the forms were coded, duplications and omissions were noted, 

/' ^ ^ iJ 

r 

On June 15 packets were sent to all program directors to refine the data 
then in hand. Directors received a cqver letter expl^ning the nature 
and purpose of the study as well as the supplements^ forms. Progranr 
directors were asked to do three things: *{a) provide information on the 
academic ability of eacf/ trainee, (b) make corrections in the list of 
trainees in their Title IV graduate training program so that the Task 
Force would have comp/ete and accurate information for each program, and 
(c) provide copies of Form 6003 for any trainees for whom forms were not 
available from the uSOE,. 

Program directors who hkd not responded by July 6 were contacted 
by telephone and encouraged to provide the requested information. ' A 
second telephone follow-up was conducted during the week of July 27. As 
a result, responses ^ere received from all but one of the 89 program 
directors; for that program, all information except data on trainee 
academic ability was available from the USOE forms, 

AnaTysjs o f Data 

Although there were 89*Title IV programs, only 88 were included 
in the data analysis s^ljice the sole trained in one program at the University 
of Georgia had dropped out of the program early in the academic year and 
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had not been replaced by another. In all, 800 appointment forms were 
obtained; these ^ere t^ien winnowed to eliminate the redundancy which 
occuirred when a^tudent left the training program and his traineeship was 
immJdiately taken over by a new trainee. Since the focus of concern in 
this study was the potential contribution to be made to American educa- 
tion and to educational research by the ^jraduates of the Title IV Research 
Training Program, the appointment form of any trainee who left the program 
(and therefore was less likely to make such a contribution) was removed' 
and his replacement was used in all analyses. This left 797 appointment 
forms for the 819 graduate traineeships awarded by the Research Training 
Branch, As there are some traineeships each year which go unfilled, the 
797 trainees included in the^present analyses very nearly refpresent the 
complete population of Title IV-supported trainees during academic year 
1969-70 (without the overrepresentatlon which would result from including 
the trainees who had been replaced during the course 'of the academic 
year). ^ - 

Wherever possible, data were coded to parallel the coding conducted 

2 

by Sieber in his earlier study of Title IV trainees. Data were keypunched 
and verified in preparation for computer analysis. 

Since data were obtained on the total population of trainees, only 
descriptive statistics were used in the analyses. These analyses, con- 
ducted so as to be as comparable as possible to analyses reported by 



2 " i' 

A code book showing in detail ±he coding of data analyzed herein 
appears as Appendix L. While this coaing parallels Sieber's, two types, 
of deviations occur: (a) some data coil^dfor the present study were 
not available for Sieber* s *study, and (Dlchanges were made in some cases 
where it was felt they represented improvements over Sieber* s codes. 



Sieher, comprised frequencies, percentages, means and standard deviations. 
The analyses went beyond the pattern established by Sieber in that 
raster's de^jree and doctoral programs were analyzed separately. In 
addition, terminated and contfnuing programs {described later) were 
analyzed separately and are compared in later sections." 

Computer programs used in the analyses include the University of 
Colorado LER Tally-X, BJjID OlD, BMD 02S, and .programs specially written 
for these analyses, ^ ' . ^ 

comparisons Used in the Analyses 

Two types of comparisons will be drawn in the sections which follow. 
First, where the 'data are comparable, similarities and differences 'in the 
characteristics of the (Sieber) 1966-67 trainees and the (Task' Force) 
1969-70 trainees will be examined. Data from other relevant studies will 
also be introduced and examined where they are comparable. 

A second^t of comparisons will be made within the 1969-70 popula- 
tion itself* Thesfe internal comparisons bedame desir|iDle when *the ^Research . 
Training Branch, in March, 1970, made the first change in the original 
lineup of 89 graduate training programs by notifying 29 directors that 
support for their training programs would be discontinued at the close of > 
the initial five-year support period and nine other directors that a similar 
decision might be made about their programs after one ^dditional year of 
support* One comparison, then, is between the characteristics of trainees 
in the 51 continued programs and the characteristics of trainees in the 38 
discontinued and probationary programs. The focus of this first Internal ' 
comparison is upon the extent to which the decisions made by the USOE will ' 



All comparisons reported tn this paper are judged, on the practical 
'significance of the differences observed between programs. 
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■likely result in better programs and trainees as indicated by generally 
accepted indicators of quality {e.g., level of talent). i 

A second comparison, within the 1969-70 population was prompted by 
the -^exigencies which resulted when the Fiscal Year 1971 appropriation to 
the Research Training Branch was reduced from $6,25 million to $2.0 million.^ 
A^^ a consequence .of this reductirfi, and the extreniely terse justification 
offered by the^Senate for^ its action . * the present programs are 
unimpyiessive'*), consideration was given to eliminating all or a m^jor , ^ 
porti(jn of the 51 remaining graduate training programs . ^/in order to 
examine the effects df such an action, the writers ^elected (on the basis 
of. their reputational -quality) 37 programs which moiit Informed observers. - 
would agree were illustrative of the better training being-offered within - 

overall Research Training ProgramT* A sepa^ate run was made of th^ 
(^laracteri sties of the trainees in these "be^tter" prograjus in order to . ^ 
allow comparisons between them and trainees in (a) the 89 original programs, 
* t^>)'*tbe 51 programs which were scheduled prior to tfje appropriations' 
fcut --^ ^be>continued, and (c) the 38 programs whiib were either' disx:orftinued 



or placed on probationary status. The reader should recog^i^jthat the 
characterisctics of the trainees in the "better" programs are represented 
in the characteristics presented for the 89 original and 51 continuing 
programs, so there is overlap in the comparisons reported herein. The., 
focus of this examination is upon (1) the extent to wliith a further 
reduction in the number of training pro'g rams would result in a commensurate 
inci;ease in quality, and (?) the dimensions of/the talent loss which wolild 
-^cunif a2i of the current graduate trainirw programs were discontinued ' 
before the current trainees had a chance to complete their preparation. 
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* The Title Itf Research Training Programs 

Before comrnencing a description of the characteristics of trainees 
Jj3 the 89 Title IV training programs, the^reader may find .it helpful tor 
have some detail about the prograins thejnselves* Since few chanqjis wore 
made with respect to the^sites of participating programs between t'lie tiiiie * 
Sieber examined them in 19fe6-67 and the present), much of what follows 
is excerpted from his. 1968 report* 



Insti tutiona? Setti ngs 

'^Sieber (1968, p. 14) found the instituti 



graduate training programs to be chiefly settings, of professional educa-' 



onal settin9s of Ihe 89 



tion: 



University departments of education 70 

Local or state school systerns -10 

Liberal ^rts and sciences departments * 5 

Nonuniversi ty and nonschool settings 

* • 

Total ' • ' 89 



78 
11 

6 

_ 5_ ■ 
100 - 



Forty percent of the .graduate programs entailed interdisciplinary training. 



A more detailed description of the Title IV programs, along with 
a critical analysis of some of their features, is presented in Technical 
Paper No. 16 in the AERA series. That paper also tracer, the evolution of 
Title IV as an integral part of the Elementary and Secondary Education - 
Act of 1965. , . 

5 ' * 

A listing of the 89 Title IV training programs for 1969^70'^may be 

found in Appendix L, pp. 2-4. * ' " 
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Regional Distribution . 

Siieber reported the regional distribution of graduate trainees 
within procH^ams and compared it t6' (a) the distribution of prapticing 

T ] 

educational researchers, as reported by Bargar, et al- (1965), and 
I (b) the distribution of public school pupiTs at that tjme. One-of his' 
major findw^^s was that the distribution^ of trainees in all TitTe'l-V 
•prog?!^hs^ak^ togethery- more closely conformed to the distribution ''of ' 
.public school errQ.llment than to that of educational researchers; among 
graduate trainees, howevefr, the distribution was more like that of 
t researchers tttan public school students^ as shown in the following table 
{Sieber, 1968, p. 12), ^ 

^ : / Table 4.1 ^ ' . 

Regional Distribution of Graduate Trainees, 
Researchers-at-Large and Publdc School Enrollment (FaLf, 1966) 

r— ' : V ^ 



Region 



Graduate 
Trainees 



New Eng>and ^ ^ 
Middle Atlantic 
East North Central 
__Jfes-t--ff6r^th Cen^il 
South Atlantic 
East South Central 
^ ^West South Central 



Researchers- Public School 
^ at-Large ' ^ ^ Enrollmen.t 



Mountain 
Pacific 



N = 



11% 

21 
7 

13 
, 5 
6 
5 
8 

100% 
(•774) 



. ' 6% 
23 

■ 23 ■ 

7 • 
14 

■ 2 
4 

■ 'V 

14 
(3,910) 



16 

21 
9 



14 
7 

10 
5 

13 



lOOX 
(41,700,000) 
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Research Quality of the Institutions 

' Various Mists have been derived of Institutions in which the 
educational research conducted and/or research training received is 
considered to be of unusually high quality. These* lists arai based 
on amoant of research funds received, the quality of research * 
produced (as judged by knowledgeable peers), the research productivity 
of students trained in the\r progrW, and on various combinations ^ 
of these and other measures. ' On/the basis of the lists he*^use^, 
Sieber concluded th^t the traimng Rrograms were located in univer- 
sities that praised the best contributifin to research training. 

Specifically, Sieber (19u p. 15) note<^ that of ttte.20 "best" 
schools in 'Keni stone's- (1959) scale of university qu^ality (see' 
Appendix L, p. 5), 15 held Ti^le IV research' training grants. This' 
analysis was carried a step further in the present study, to the 

number of trainees being supported i» programs located at Keniston's 

t , * ' ' ' 

"best" universities, the results of that analysis are depicted in ^ 

J^bVe 4,2. \ 

Sieber and-kazarsfeld (1966) developed a list of 22 schools-of 

^education which were named in 1964-65 by deans and research coordina- 

t(^rs in education as producing the most competent and worthwhile 

Research (AfJpendix L, p. 6). Seventeen (77%) of the 22 schools of 

j^ducation hold a Title IV research trainii^g grant. The proportions 

Qf 1969-70 trainees in programs located in the schools of education 

identified by Sieber and Lazarsfeld appear in Table 4.3. 
* 

Worthen (1968) ranked 40 universities that had been identified 
in other studies ^as ins„titutions high on some educational reseaVch 
index (Appendix L, p. 6), Of these, 24 (60%)' have Title IV training 



programs. /In Table 4,4 is presented the prbportic/n of trainees 
pursuing their stydies (?) at institutions listed among the first 
ten cited by Worthen, (2) in the next thirty institutions, and 
(3) at institutions not included on the Worthen list. 
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Table 4.2 



. 'Trainees in programs Located iij Institutions 
Ranked "Best" Upon K^nis ton's Scale of University' Quality 



Doctoral Programs Only 



Category 


.. ' All 
Programs 


Continuing 
Programs 


1 

Probationary 
and Phased ^Out 
Programs 


"fetter" 
Programs 




N . 




M % 


N 


■ % 


H 1 


"Besr on 

l^onT c 't'nn * c cr^^lo 
iNciiioUUii o bUultr 


294 


45 


267 


60 


27^ 


a 13 


201 b6 


Not •iBest" on ^ 
Keniston*s scale 


357 


55 


179 


40 

* — — „ 


178 


87 


157 44 


Subtotal 


651 


100 


444 


100 


205 


100 


358" 100 


c ^ 

G 




Subdoctbral Programs Oiily 






"Best" on ' / 
kenistori's scale 


■ 3 


i 

2 


0 


0 


3 


2 ' 


0 0 


Ndt '^'Best" on 
Keniston's scale 


t31 


90 


14 


100 


117 


\ 89 


9 100 


Not Appall cable 




8 


'0 


0 • 


12"^ 


9 


0 - 0 


Subtotal 

* 


146 


100 


14 


100 


132 


100 


9 100' 


TOTAL 


797 




460 ' 




337 




3^7 



(see footnotes .on next page) 
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Table 4,2 footnotes 



\ ' Perhaps an illustration^^of how the data in the tables can be * 
"read" would be helpful. The/reader will recall the Task Force's 
interest in the comparison between the' 51 continuing programs and the 

'38 probationary and discon^nued- programs was whether the indices of^ 
quality were improved or Heightened as*a consequence bf the Research 
Training- Branch's action/ The data in Table 4,2 mayXtherefore be 
exan^fffed thusly: A higher proportion of the continuing doctoral " 
pro^r^ms (60%) are inauded on Keniston*s scale of university Quality 

, tha;i were included in the original group o!f doctoral programs (45%), 
siy tlje^ effect of th4 USbE action^on this particular index was to 
r^lsTe'the overall/guality of the (diminished in size) training program. 

Njhe" Small number of trainee"^ in doctoral programs a,t "best" universities 

, wh4cfii were placed on probation or discontinued (27 trainees) suggests 
that on this particular index the USOE action dy not result in -a 
significant /proportion of high quality trainees oeing "carried away 
with the bath water." ; 

Th9 Task Force's interest in examining the relationship between 
the 51 continuing programs and 37 "better" programs was twofold: 
to see whether further reductions would result in a commensurate 
increase in quality — at the expense of numbers of trainees, of / 
course; and, secondly, to assess the loss of talent which would ^ 
result frofn summary elimination of all of the current graduate training 
programs* The data in Table 4.2 indicate that a ^further deduction in 
programs would result in a decrease .(frcwi 60% to 56%) in the 
proportion of doctbral trainees being prepared at Keniston*s "best" 
universities. The data in Table 4*2 do not treat directly the talent . 
loss which would result if all training programs were discontinued 
(later tables will do so) but one may infer that loss of the 60% ' 
of trainees enrolled in the "best" universities would constitute a 
talent loss of some magnitude* ^ 

y/^hese trainees are from a program administered by a group of 
univpsities and school districts; therefore the program could not 
be ranked on institutional quayty* ^ 
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Table 4.3 



Trainees in Programs Located in Schools Education 
'Identified bySieber and Lazarsfeld for the Quality of Their Research 









Doctoral Programs Only 






Category 

t 


All 

.Programs 


Continuing 
Programs 


Probationary 
and Phased Out 
Programs ■ 


"Better" 
. Programs 


t 


N % 


N 


% 


N ' % - 


N 


School IS'on 
list 


316 


49 


283 


63 


33 


16 


235 66 


School is not 
on list 


326 


50 


154' 


35 


172 


84' 


114 32 


Not Applicable 




1 




2 


-0 


0 


9^. 


Subtotal 


651 


100 


446 


100' 


205 


100 


358 101 



Subdoctoral Programs Only 



School is on^ 
list 


12 


8 


9 


64 


3 


\ 


9 


' 100 


Schd^l is not 
on (Ust 

Not Applicable 


122 


84 
8 


5 

«.o 


36 
0 


117 
12^ 


89 
9 


0 

{' 
0 


0 
0 


Subtotal 


.146 


100 


14 


100 


132 


100 


9 


V 

100 


TOTAL 

= _^ 


797 




460 




337 ' 


0 


367. 





'These trainees are in a program which is not located in a school of 



education. 

'^These trainees are from a program administered by a group of universities 
and school districts; therefore the program cou\d not be ranked on institu- 
tional quality. 



Table 4.4 



^<^\Trainees in Programs Located in Institutions 
Cil^d by Worthen for the Quality of Their Educational Research 





Doctoral Programs Only 


r 

Category 


\ 

All 
Programs 


Continuing 
Programs 


Probationary i 
and Phased Out 
Programs 


"Better" 
Programs . 




N 


% 


N 


% 


N 


% 


N : 


Among top ten 


230 


35 


208 


47 


22 


n 


16b 

i 

129 


46 


Amona next 
thirty 


181 


28 


152 


34 


29 


14 


36 


Not on list 


240 


37 


86' 


19 


154 


75 


64 


18 


Subtotal 




100 


446 


100 


205 


100 
-* 


358 


100 








Subdoctoral Programs Only 






\ 

Among top ten 


3 


2 


0 


0 




2 


0 


0 


Among next 
thi rty 


9 


6 


9 


64 


0 


0 


9 


100 


Not on list 


122 


84 


* 

5 


36 


117 


89 


0 


0 


Not Applicable 


12^ . 


8 






12^ 


9 






Subtotal 


146 


100 


14 


100 


132 


100 


9 


100 


TOTAL 


797 


* 


460 




337 




367 





These trainees are from a program administered by a group of universities 
arid school districts; therefore the program could not i)e ranked on institu- 
tlonafl quality. 



Recruitment Practices and Admissions Standards . 

Each year usually ^t the time when funds are requested for 
the following year's operation — the director of each Title IV 
training program submits a progr^s report to the U, S. Office of 
Education. Jhe 89 progress reports submitted with 1969-70 proposals 
serve as. the source for information presented here and in the 
following sections^ 

Eighteen progress reports either gave no information regarding 

4 

entrance requirements or gave information which was too general to 

classify. Of the 71, programs which provided usable Information, 

53 either require or recommend that Graduate Record Examination 

scores (verbal and quantitative) b^ presented by applicants. Most ^ 

of these do not specify minimum scores for acfcnission; for those which 

do* the minimum acceptable total GRE score ranges from 900 to 1250 

vri th a mode of 1000 (10 of 19 programs). Examination of the progress ^ 

repoVts indicates that in most cases the GRE requirement for admission 

Wihe Graduate Research Training program is the same as that for 

J. 

admissionvto the institution of which it is a part (or to the school 
or college of education). 

The Miller Analogies Test is recommended or required for admission 
to 30 fif the training programs- Other standardized tests used by one 
or more of the programs exanjpned include the National Teachers Exam, 
the Doppelt Mathematical Reasoning Test, the G^ Advanced sections 
(either in education or in another area of the applicanlH..chtte?j:)g) , 
the STEP Writing Test, the Cooperative English Tests, and the Watson- ' s 
Glaser Critical Thinking Appraisal. 
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Of those programs whicn provided usable information aoout^ 
their entrance requirements, nearly all specified that a student's 
undergraduate and graduate scholastic record is^important in the 
selection process* Most require a 3.0 undergraduate grade point 
average (on a 4.0 scale) and a 3.5 graduate GPA. It should oe 
emphasized, however, that only 19 of the 89 programs gave specific 
information on the question of GPA; therefore, it is not known V^her 
the 3.0 and 3.5 standards are maintained by roost programs. ^ \ 

- Eight of the programs studied specifically mention that 
teaching experience is required for admission or that preference is 
given to applicants who tiave such experience* It cannot be determined 
from the progress reports whether or not the remaining PI programs 
require teaching experience for admission. Nonetheless it is- 
interesting to note that of the 71 reports which furnished Hnforma- 
tion of some kind regarding admissions requirements, only eight 
indicated the requirement of teaching experience. If this is an 
accurate portrayal, it seems likely that the majority of programs 
will produce persons who continue in research careers. Sieber and 
Lazarsfeld (1966) found the requirement of teaching experience 
dysfunctiooal ifl_j)regarinc[ fiersgns JFg^^ 

, About half the Title IV programs eUgage in active recruitment 
for prospective, trainees. At^the very least, this involves the 
preparation and distribution of flyers or pamphlets describing the 
particular Graduate Research Training program. Several program 
administrators go beyond this, using contacts in other areas to 
publicize their programs. 
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Type of Program * 

In Chapter 3 above, areas of coriipetence in research and research- 
related activities were listed as derived from 1968 AERA employment 
service forms. This list provided the categories of program type 
and course type which are used as headings, in Tables 4.5 through 
4.7. Not all of the areas of corrpetence were -ased in deriving the 
categories for th^se tables, however; some were combined and several 
were omitted. One additional category — mathematical statistics 
was created to descrfbe two programs which emphasize statistics but 
not in conjunction with research methodology and design. The ^ ^ 
resulting categories describe fairly broad areas which encompass all 
of the programs discussed here. 

Each progress report was examined for any information which 
would aid in identifying the type of program. In some cases this 
could be derived from the stated objectives of the program; in others, 
it was necessary to consider course requirements, practicum experiences 
and miscellaneous statements in order to determine the program type^. 

In Table 4.5, the number of doctoral and subdoctoral programs 
in each ''type" category is indicated. Since sme of the doctoral 
" progiWiT have ^vera1"5^parate^iaTid litstinct-ardls-of-emph^i^^^^ - — 

doctoral total is greater than the actual nuirter of doctoral programs. 
X It was difficult to be definite about the type of several of 

the subdoctoral programs since they seem to be most accurately 
described as surveys of research or research- related areas. Thus the 
classification of subdoctoral programs is more ambiguous than is that 
of the doctoral programs. 
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Classifi cati< 
Programs by 



Table 4,5 ' 

of Title IV Graduate Research Training 
^pe " Doctoral and Subdoctoral Levels 



Type of Program 



Research methods, des<gn,\^tati sties 
Educational development 
Educational diffusion 
^valuation and/or measurement 
Computer methods, utilization 
Psychology * ' 

A(inin'i Strati on 

Curriculimi (including specialized 
subject areas) 

Guidance and counseling 

Vocational education 

Special education 

History and philosophy of education 
Other social and behavioral sciences 

/ 

Mathematical statistics 



Number of Programs 
Doctoral 'Subdoctoral 



17 
4 
1 

9 
3. 
15 
8 

22 
2 
2 
2 
6 

15 
2 



4 

0 
1 
4 

0 . 
0 

0- 

0 
0 
0 
0 
0 
0 
0 
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It can be seen that most programs on th^doctoral level 
emphasize (1) curriculum, (2) research methodology, design and statis- 
tics, (3) psychology and (4) other social and behavioral sciences. ' 
It should be pointed out that the majority of programs listed under 
the curriculum beading were, in fact, programs in specialized subject 
areas (e.g-, math education or reading)* All except one of the \ 
subdoctoral programs fell into two categories: (1) research methodology. 
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design and statistics, or (2) evaluation and/or measurement. As It 
was pointed out abo^e^ however, so^me of these' might more accurately 

be described as surveys of research • 

* » 

Course Requirements ^ , , * 

The headings used in Table 4.5 to'fdentify the typ^ of program 
were also used to classify required and elective course^ Usable ( 
information listing or describing courses was fpund in 67 progress 
reports, but some of those 67 did not provide complete information ^ 
(e.g., they described only new course requirements established since 
the previous report). Still, several generalizations may be. made 
regarding the classroom preparation of Title IV trainees. \ 

Almost every program requires coursework in^ research methods^ 
design and statistics, and most appear to require at least three 
or foujr courses in this area. Since these are research training 
^programs,' this requirement is not surprising. Nevertheless, It seem§/ 
to be one requirement common to the prograriE* 

Table 4.6 shows the count of doctoral and subdoctol^al programs 
(of those furnishing specific information) which have required courses 
in each area. This table contains no data on how many courses are 
required in each ar^ea; It Identifies only the types of courses required.* 

The following should be noted in interpreting the informati'on 
In Tabl)^ 4.6. Courses' concerned with testing (test construction, 
explanatidn of standardized tests, etc.) were usually placed in the 

^ Technfcal Paper Na. Ij6, which deals in more detail with Title 
IV programs, contains information on the relative emphasis of various , 
classroom experiences* 



."evaluation and/or measurement" category. Most of the 35 programs 
noted in the table in this category require cburses in testing as 
opposed to evaluation £er sie . Courses classified as "other social 
and behavioral sciences" were for the most part in sociology. Those 
in "other disciplines*' were diverse — from art. to mathematics. 1 



Table 4-6 

Types of Courses Required in Title IV Graduate Research 
Training Prograrfis — Doctoral and Subdoctoral Levels 



» 

* 

* 

• 

Types of Courses ^ 


Number 
Doctoral 


Programs 
-Subdoctoral 


Rp^parch mpthod^. de^ion. statistics 


58 


. 9 


F HijPA ti nn;)1 dpvplnnmpnt 




0 


Educational diffus.ion 


2 


0 


Evaluation and/or measurement 


27 


8 


Computer methods, utili^tion ^ 


^ 27 


5 


Psychology 


,, 27 


5 


Administration 


n 


1 


Curriculum (including specialized^ . 
subject areas) 


' 20' 


1 


Teacher education 


\. 6 ^ 


0 


Vocational education 


1 


0 


Special education 


1 


0 


History aod philosophy of education 


20 


1 " 


Other social and behavioral sciences 




2 


Other disciplines 


t 

7 


0 

, _y 



/ ■ 



Fewer progressj:^ort;s gave information on elective courses 
For thej^S that dicf, Table 4.7 shows how their courses are classified. 



Table 4.7 

Types of Elective Courses in Title IV Graduate Research 
Traihing Programs — Doctoral and Subdoctor^il Levels 



Types ofrCourses 



Number of iPr^o^njs 



. Doctoral 



Subdoctoral 



Research methods, de$ign; statistics 
Evaluation and/or measurer^ent 
Corfiputer methods, utilization 
Psychology 
Adnini strati on 

Curri cul urn ( i ncl udi ng sped a 1 i zed 
suWect areas) 

Guidance aod counseling 

\ Teacher education* 

Special education 

History and philosophy of education- 
Other social and.behayioral sciences 
other disciplines 



10 
12 
10 
14 

7 

9 
4 

2 
4 
7 
9 
4 



1 

0 
1 
0 
0 



\ 

0 



0 

' 6 

0 
. 0 
0 

e 

.0 



Practicum Experiences ' • ^ ' , ' 

* ♦ 

fhis was ^e most difficult section of Jthe progress reports' 
to classify because tMre was' such diversity in the way program 
directors chose *t)^^esiri be this aspect of the programs. Eighty* ' 
progress reports provided enough information for at least a partial 
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descViption of their practicum arrangements, • 

Two kinds of information were extracted from the progress 

* * . - • 

ireports. First, it was desired to* know in what setting the practicum 

( ' ' * ^ • - - ^ 

experience occurred for each program. This information is given 

1n Table 4.8. The rtumbers in this table reflect the fact that 

# 

Xl) niany programs place some trainees in one kind q,f practicum setting 
and some in another, (2) some trainees have more than one practicum 
and (3) some trainees have a^ practicum which places them in two 
settings jointly (e.g.,; a university education department and a 
school oistrict). It can readily be ^s6en that the vast majority 
of practicum experiences take place in university settings and in 
schools and school districts. ' * 



V. 



Table 4.8 " 

Institutional Setting of Title IV Program Practicum.* ^ ' ^ 
Experience ~ Doctoral and Subdoctoral Levels 





Number of Programs . - 


Institutional Setting^ 


' ^ ; Doctoral 


Subdoctoral 


' » ' ; 






University - Education Department 


44 




University Other Departments ] 


' ^ 37 


■ ' 4. ... * 


R&D Centers ; 


7 


1 


Regional Educational Laboratories 


' • 11 


0 ^ * 


Schools and School Districts 




8 ' . 


State Education Departments 


1o 




Independent Research Agencies 


' ^ 19 


? ', 


Federa,] Agencies 




.2 ^ 


Industry '! * ^ 


. 10 


2* 


Professional Education Associations 


4 


• 0 ' 



'These Institutional Settings were first listed in Technical 



Paper No. 2 in the AERA series 



The second kind of infonnati<fn concerned the type of assign- 
ment Jie Id. by the trainee during his practlcum. Worthen and Roaden ' 
•(1970) found that for research assistants In educational research 
.generally,, assignment to. an Indlvldtial faculty menfcer but n£t in 
^relation tt a research bureau or on one specific project 1^* 
-positively correlated with later research productivity. The •Title 
IV training* program reports were therefore examined for Information 
on 'this aspect of the pi'acticum- * ^ 

Usable data were obtained from 53 progress reports. Within ^ 
each program, all practicum arrangement's mentioned in the* report 
were considered together^nd a determinatit)n made as fo the /usual 
patteh) of experience. This pattem'of practicum assignments for the 
53 programs is shown in Table 4.9. It appe^irs that at l^ast '^three- 
quarters of the practicum assignments are either to a single faculty 

r 

member working on one project or to a research lab or bureau in which 
many senior resear^hers^^ and many projects are involved. 

^ J Table 4.9 ' . ' 

Type of Assignipent in Title IV Program Praoti.'m 
Arrangements — Doctoral and Subdoctoral Levels 



Number of Program^ 
Type of Assignment 'Doctoral , Subdoctor^al 



1 researcher/ 1 project 


17 


'V 3^ '•• 


1 researcher/ more than 1 project- 


6 


' 2 ' 


more than j researcher/ ^project 


5 


0 • . 


more than 1 researcher/ more than 






1 project 


18 


^ 2 \ 



It jwas also indicated in twer)ty-three progress reports ^that soiik>'' 

sort of sernlRarfor trainees was held; In son»e cases these w^re 

scheduled weekly, while in others th§y were held only sppradioally. 

' - . ' J . 

Topics of the seminars ranged from discussions of methodology in 

specific experiments, to reports of research being conducted by 

Individual faculty members or trainees or problems of school 

administrators in specific areas. Most program administrators j[and, 

by inplication, the trainees) viev/ these seminars as valuable, both ^ 

for the learning experience they offer and for the opportunity they 

present for trainee and faculty interaction. 

The "Ripple Effect** of Title Training Programs 

Data on nonstipend Title IV trainees (trainees associated with 
the program but not supported by Title IV funds j^/ere collected in the 
present study to determine how many such trainees exist and what lev^l of 
talent they exhibit. 

Ther-e were found to be 113 nonstipend trainees in^^the 89 programs. 
On indices of academic talent, they "tell somewhat below regular trainees 
on Miller Analogies Test and Graduate Record Examination scores, as follows 

MAT Scores GRE Total Scores 





Mean 


Jl_ 


Mean 


_2L 


Nonstipend trainees 


58.92 


26 


1,169- 


59 


Regular trainees. 


64.78 


309 


• 1,205 


•473 



On graduate grade-point average, the nonstipend group mean of 3.67 on a 
4-point scale (N=88) was slightly higher than the regular group mean of 
3*62 (N=618), It should be noted, however, that the very great diffe%nce 
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n * 

V 

In the sizes of the two groups makes direct comparisons between them 
tenuous at best. 

It Is doubtful if a large proportion of the nonstipend group would 
be receiving systematic training in educational research in the absence 
oT^the Title IV training programs. If this assumption is correct, Title IV 
might be viewed as having the "ripple effect" of providing training for 
considerably more trainees than the approximately 800 supported by Title 
IV funds. 




\ 




\ 

\ 



Primary Indicators of Trainees' Potentiul 
P roductivity and Contribution 

Certain of the characteris'tics of trainees are considered to be 
stronger and/or more direct indicators of potential pjfoducti vity and 
contributipn than others.. Fleury and Cappelluzzo (1969) found that the 
literature on research training indicated there were five factors which 
should be considered when developing recruitment procedures and entrance 
examinations for prospective educational researchers: (1) age at the 
doctoratef (2) level of student talent, (3) previous teaching experience, 
(4) academic background, and (5) undergraduate and graduate grade point 
averages.^ Data on these five factors will be presented in this section 
for the benefit of th^t large proportipn of readers whose interest will 
be satisfied without proceeding further. Data on additional, related ^ 
variables will be presented In the section which follows for those who 
are interested in probing further Into the subject. j 

In brief, the reader will find the following selected comparative i 
data reported in this section; 

J " 1966-67 Trainees 1969-70 Trainees 

1. Mean age at graduation from 

doctoral training programs ' 31.2 years 31.1 years 



Fleury and Cappelluzzo found these entrance requirement variables 
were not* effective predictors of the relative success of the trainees in 
completing their programs; in fact, they predicted success only about 
fifteen percent of the time* fJonetheless , logic supports the conclusions 
of investigators in the realm of research training that these variables 
must be considered in recruiting and training productive professionals. 
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Z. Level of talent of 1969-70 doctoral trainees as compared to first- 
year graduate students outside education. 



Rank 


Discipline, Profession, or Set of Trainees 


Mean MAT Score 


1 


Applicants for Psychoanalytic Training 


68.4 


2 


Trainees in continuing doctor-al programs 


66.8 


3 


Trainees in "better" doctoral programs 


66.7 


4 


Trainees in all doctoral programs 


65.9 


5 


Psychology 


65,3 


6 


Medicine 


, 64.5 


ll' 


Business Administration 


54.1 


12 


Engineerir>g , 


53.5 


13 


Social Work 


49.6 



196 6-67 Trainees 1969-70 Trainees 

3, Lapse of less than five years 73% 86 

before entry to graduate training 

4* Undergraduate jnajor field 



Education 


Buswell's 1964 
Education 
Doctorates 

30% 


1969-70 Doctoral 
Trainees, All 
Programs 

26,;. 


Social Sciences 


23 


39 


Natural Sciences 


15 


' 23 


Humanities 


18 


9 


Other 


13 


2__ 


TOTAL 


' 99% 


100% . 



5* Graduate grade-point averages, 1969 doctoral trainees 

I, 

All Programs 3.63 
Continuing Programs 3.65 
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Age ' V ^ -..^ 

Age at time of entry i nto training. A nuiiter of investigators 
have held that educational researchers who complete their doctoral 
programs by age 32 are more productive than older doctorates (e.g., 
Buswell, et al-, 1966; Sieber & Lazarsfeld, 1966; Millikan, JS€f). 
Entry age data gathered on the 1966-67 trainees by Sieber and the 
1969-70 trainees in the present study are presented in Table 4 JO. 

Table 4.10 

Age at Time of Entry into Title IV Gradigite Training 
Program, 1966-67 and 1969-70 Trainees 



Doctoral Trainees Only Master's and Others Only 



Age 
Category 


1966-67 


1969- 


■70 


1966- 


■67 


1969- 


•70 


N 


% 


N 


% 


N 


% 


N 


% 


19 - 24 


178 


31 


167 


26 


53 


39 


45 


32 


25 - 29 


173 


30 


250 


39 


27 


20 


41 


29 


30 - 34 


117 


20 


121 


19 


18 


13 


23 


16 


35 - 39 


64 


11 


61 


10 


18 


13 


8 


6 


40 - 44 


37 


6 


26 


4 


14 


10 


12 


9 


45 - 49 


12^ 


2^ 


7 


1 


6^ 


5^ 


6 


4 


50 - 54 






3 • 


' 1 






5 


4 


Total N • 


581 


100 


635 


100 


136 


100 


140 


100 



^Sieber's last category was 45-I-, so these results compare to 
the last toyo age categories for the 1969-70 trainees. 



Sieber (1968, p. 78) noted that a distinct contribution of the 
Title IV Research Training Program was to lower the age at receipt 
of degree by about seven years. Further progress has been made since 
1966-67 in recruiting younger trainees. In 1966-67, 61% of the trainees 

ER?C , . . - 



251 



mre 29 years of age or younger upon entry to the doctoral program; 
in 1969-70 there were 65% of the trainees of this age at time of entry. 
The mean age upon entry to the graduate training programs was: 



DNoctoral programs 
Subdocloral programs only 
All graduate programs confined 



1966-67 
Trainees 

29.0 years 

-29.2 

29.1 



1969-70 
Trai nees 

28.5 years 

30.0 

28*8 



Age at graduation . Data were also gathered on the 1969-70 trainees' 
age at the expected date of graduation^ as shown in Table 4.11. 

Table 4.11 

Nui*er and Percent of 1969-70 Trainees in Various Age 
Categories; at Expected Date of Graduation 



Age Category 



Doctoral Trainees 
Only 



% 



Subdoctoral Trainees 
Only . 



N 



% 



19 - 


24 years 


13 


2 


19 . 


31 


25 - 


29 • 


285 


46 , 


34 


36 


. 30 - 


34 


182 


29 


10 


n 


35 - 


39 


96 


. 15 


4 


4 


40 - 


44 


27 


4 


7 


8 


45 - 


49 


16 


r - 

>i 




3 


50 - 


54 


2 


6 


6 


55 - 


59 


1 


>i 


1 


1 




TOTAL 


662 


99 


84 


' 100 
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The projected mean age at graduation from the graduate trainint? 
programs was computed for the 1966-67 and 1969-70 populations of trainees: 

Projected Mean A ge at Gra duati on 
1966-67 Trainees 19 69-70 Trainee s 

Doctoral programs only 31,2 years ^ 31.1 years 

Subdoctoral programs only 30.3 30.4 , 

All graduate programs contin^d 31,4 31.0 

Eighty-five percent of the 1969-70 trainees will likely complete 
their doctoral programs prior to the mean age (36,6 years) of liuswelTs 
1964 Ph.D. recipients and virtually all will, complete their programs prior 
to the mean age of the 1964 Ed.D. recipients (39,0 years)- 

Time lapse between entry and expected date of degre e. Tne mean lapse 
of time between entry to the graduate program and expected receipt of degree 
computed by Sieber was quite similar to -that found for the 1969-70 trainees. 

Lapse of Time Between Entry and 
^ ' Expected Time of Deg ^ree _^ 

1966-67 Trainees 1969-70 Traine es 
Doctoral programs only 2.4 years 2.4 years 

Subdoctoral programs only 1.1 1.2 

All graduate^progrmas combined 2.2 2.3 ^ 

Level of Student Talent 

, In their study of the ways in which American educational research 
is organized, Sieber and Lazarsfeld (1966) concluded that the level of student 
talent is probably the most important factor which should be considered 
when recruiting prospective professionals. Fleury and Cappelluzzo (1969) 



found in their questionnaire study that training directors generally accept 
scores of standardized tests as indicators of the level of student tah^nt 
The returns to -thei r questionnai re indicated that, in 1966-67, 49.4 of the 
trainirrg directors used the Miller Analogies Test (MAT) as a selection tool 
^ .and 76*5% used the Graduate Record Examination {GRE)* 

In Table 4-12 are presented the data on the MAT and GRE scores of 
1969-70 trainees- 

Some con^arative data are presented in the next few pages to help • 

r 

the reader interpret the mean HAT and GRE scores presented in Table 4.12* The 
reader is cautioned, however, that the comparisons^ are between scores uf a 
select group of scholarship-level education students (Title IV trainees) 
and scores achieved by the full range of students in -other professions and 
the disciplines* Under these circumstances, the education trainees snould 
be expected to compare favorably, and they do* What is of interest is the 
extent to which they score higher than students in the other professions 
and the disciplines. Of equal interest are comparisons between the level 
of talent among trainees in probationary and discontinued' programs and the 
level of talent among trainees in continuing programs* ' ^ 

Mean MAT scores for the various groupings of Title IV graduate training 
programs are compared in Table 4J3 with the percentage of students-'in other 
professions and the disciplines who scored lower on the same test* 

Only 13 percent of education graduate students (excluding educational 
administration students) in doctorate degree-granting institutions scored 
^ ^ 0^ better than the average trainee in a continuing or "better" 
doctoral program. Only 14 percent scored as, well as or better than the 
average trainee in all of the doctoral progratfc combined. Sixteen percent 
scored as well as or better than the average trainee in the probationary and 
discontinued doctoral programs, 

Er|c n ' 
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Table 4.12 

Mean Scores on the Miller Analogies Test and 
Grcfcduate Record Examination by 1969-70 Trainees 



Category 



Doctoral Programs Only 



All 
Programs 



Mean 



Continuing 
Programs 



Mean 



Probationary 
and Phased Out 
Programs 



Mean. 



MAT scores 
N 

GRE Verbal 
N 



65.92 
(260) 

610.33 
(414) 



N 

GRE, Total 
N 



MAT scores 

N ., 

GRE Verbal 
N . 



(414) 

1225.53 
(414) 



- 58.78 
(49) 

536.61 
(59) 



N 

GRE Total 
N 



(59) 

1057.46 
(59) 



14.79 



99.11 



GRE Quantitative 615.92 102.12 



154.49 



66.84 14.31 
(160) 

624.43 96.70 
(300) 

621.53 100.61 
(300) 

1246.28 146.89 
(300). 



64.43 
(100) 

573.25 
(114) 

601.14 
(114) 

1171.47 
(114) 



14.72 



96.14 



105.00 



161 .22 



"Better" 
Programs 



Mean 



66.69 
(126) 

622.68 
■ (239) 

631.04 
(240) 

1254.06 
(239) 



Sub-dbctofal Programs Only 



15.02 



-37.68 



GRE Quantitative 520.51 86.33 



147.27 



15.32 



96.39 



63 



139.73 



60.78 


13.8fe 


58.33 


15.40 


60.78 


13.86 

V 


(9) 




(40) 




(9) 






ND , 


' 536.61 


87.68 


no;. 


NO 


ND 


ND 


(59) 




NO 


' ND 


ND 


ND 


520.51 


86.33 


NO 


ND 


ND 


ND 


(59) 




^ ND 


ND 


ND 


ND 


1057.46 


147.27 


/ ND 


ND 


ND 


ND 


(59) 




/ ND 


ND 








/ 

/ 







ND indicates No Data 
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Looking outside of education, with the exception of students . 
in psychology and medicifte, and applicants for psychoanalytic training* 
the trainees in the continuing and "better" doctoral programs compared^ 
almost as favorably with. students in. other professions and the dis^ciplines 
^ as they did with students in education* Their level of talent wtfs 
also acceptable when compared to that of students in psj^chology and 
medicine, and th«it of applicants for psychoanalytic training (SOth, 57th,- 
and 40th percentiles, respectively). 

The master*s, specialist and other trainees in subdoctoral 

Title IV training programs also compared acceptably to the rank and 

f 

file of students outside psychology and medicine, and applicants for 
psychoanalytic training. 

Another, perhaps easier, way of comparing the level of talent/ 
is through a simple ranking of the mean scores for the various norm 
groups- For the MAT, the norm group^ rank as shown in Table 4.14, 

^ The mean scores of various groupings of the Title IV trainees 

on the verbal section of the Graduate Record Examination are compared ; 
with the scores ''achieved by other norm groups in Table 4, 15. Of 
particular Interest are the percentile scores for students in "All 
Fields" and in the social and behavioral sciences, e.g., for trainees. 
An the continuing doctoral programs: All* Fields (69th percentile) , 
government (56th percentile), psychology (52nd percentile), and socio- 
logy (67th percentile)'. 

Rank ordering of the desciRlines, professions, and various sets ' 
of training programs by mean GRE-Verbal scores produces the list in 

, Table 4.16. 
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Table 4.14 / ' ^ - 

Racking *of 1969-70 Title IV Trainees and Norm Groups in other Professions 





and the Disciplines on the Miller Analogies Te^t 

: . ' / 


f 




Rank 


Discipline, Professioft, or Set 
OT iraimng rrograms 


^ ' — — ' 

Mean IJIAT 
Score 


otanuarQ 

Deviation 


N 


1 


Mppiicants Tor psycfioana 1 jr ti c training 




.16.5 


.240 


2 


tontinumg doctordl progrsnis 


66 .8 . 


14.8 


160 




"Better" doctoral programs 


\. "66.7 








All doctoral programs s 


65.9 


14. o 




5 


Psychol ogy 


65.3 


14.4 • ' 


2644 


,6 


Medicine 


64 .-5 ■ 


12.0 ' 


627 


7 


Probationary and discontinued doctoral 
programs , 


64.4 


14.7 


100 


o 
o 


Continuing and "better" subdoctoral 
programs 


60.8 








All subdoctoral programs 


CO o 

bo.p 


15.0 


49 


ih 

1 u 


rroDationary anc Qiscontinuea iuuuouturd^i 
programs 


do. J 


1 0 .*f 




11 


DUbiiicd^ MUin 1 n 1 d t rat 1 un 


1 

0*T * 1 


15.2 


303 


12 


Engineering 


53-. 5 


14,1 ' 


525 


• 13 


Social Work 


49.6 


1 R ^ 






Rehabil itatiom Counseling 


49.6 


"ICO 


yino 


15 

4 


Education in doctorate-granting Institu- 
tions (except Admin.) 


46.9 


16.2 


765H 


16 . 


Nursing ^ 


46.4 


14.2 


212 


17 


Theology 


44.6 


15.7 


1920 


18 


Education Administration in doctorate- f 
granting institutions 


44.5 

* 


15.# 


1247 


19 


Education in Master' s, degree-granting 


' 39.2 


15.4 


3604 



institutions 



• ' • Table 4.15 

risj^ 'Compar1sor> of Mean Scores of M96$-70 Title IV Trainees with Perbentiles 
.* " ' Ibf First Year Graduate Students Scoring Lower on the 

Graduate. Record Examination,, Verbal^ 



Training, 
programs 
Included 



>1ean 

GRE-V 
Score 



.Percentile of First Yea-r ^Sra^duate Studen1;s 
Scorirrg Lower on the GRE - Verbal,* 



CD 
O 

o 

•I— 

CO 







C7> 










C7> 




























C 


u 


























LU 














o 


s- 














CL 




o 










cu 




>> 






C 




o 


to 




O 




t/J 


£ 


cu 


JC 


o> 


c: 






i 


to 


u 








•r- 


no 


c 

*r- 


He 


o 


er 


to 




he 




to 


ch 


o 

•r- 


c 


0) 


o 


O) 


0) 


o 


> 


CO 








>> 


>> 


U 




o 


Fr 




r 1 


& 


fin 


•r- 
31 




1 




JZ 
Cl. 


to 


O 

I/) 


p. 
</>. 



Cof^inuing 
^doctoral programs 

"Better" doc- , 
toral programs 

All doctoral 
programs 

Probationary & 
disconJ;inued 
doctoral 
programs 

Subd'octoral 
programs 



624 
623 
610 

573 
537 



78 64 62 75 43 66, 56 59 43, 66 31 47 52 67 



68 69 



77 63 61 75 • 42 66 56 59 43- 65 30 ' 47 52 66 66 6 



59 ' 65 



73 59 57 71 37 61 52 1 54 38 60 26 41 47 61 

V > ' - . . 

62 47 .46 58 30 53 41 41 28- 50 16 29 34 51 54:56 

55 37 34 42 24 39 3^ 27 12 41 U 20 24^ 4d 50 47 




Source : The Performance 
Serv.ice, July, 



of First Ye ar Graduate Studentg'on the GRE , Educationa 
1968 ^ ^ 



^The norm population was those students 'who enroVled as full-tirite graduate students 
for the first time in Fall, 1964 in a member institution bf the Council of Graduatf* J 
Schools. Comparisons of these percentiles with more recent norm data (for 1965-68 
but simply distributed Men, ^omen, and Total) Indicated that*more recent mean scores are 
one to two points lower. ^ - : 
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, TaLlel 4.16 

Ranking of 1969-70 Title IV Trainee's land Norm Groups in other Professions 
and the Disciplines on the Graduate Record Examinationi Verbal 



Discipline, Profession, or Set 
c7Rank of Training Programs 



Mean GRE-V Standard 



Score 


Deviation 


N 




. 90 


289 


626 


93 


■"-^37 


624 


97 


300 


623 


96 


239 


oZZ 


103 


cio 


610 


99 


414 


609 


89 




^ Art 

608 


105 


146 


587 


103 


259 


586 


117 


212 


581 


104 


260 


566 


117 


280 


559 


120 


289 


556 ' 


112 


126 


547 


124 


3812 


546 


^ 134 


251 


542 


96 


77 


539 


110 


175 


537 ^ 


88 


59 


518 


T28 


163 



1 Philosophy 

2 Literature m English ^ 

3 Continuing doctoral programs 

4 "Better" doctoral programs 

5 Physics 

6 All doctoral programs ' ' 

7 Psychology 

8 French . ^ 

9 History ^ 

10 Government 

1 1 Economi cs 

12 Chemistry 

13 Probationary and discontinued doctoral 
, programs r 

14 Geology * . ^ 

15 All Fields 
16* Sociology 

17 Spanish 

18 Engineering 

19 Subcoctoral program 

20 Biology 
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Comparative data on the quantitative section of the Graduate 

Record Examination are presented InTable 4J7.The Title IV trainees' 

\ 

performance on the quantitative section compares less favorably than it 
did* on the verbal section to students in mathematics, the natural sciences 
(biology, chemistry, geology, and physics), and the mathematics-oriented 
content areas (economics and engineering by 50 to 57 percentiles in 
the latter easel). In the 'humanities and social and beha^fioral science 
areas (French, government, history, literature in English, philosophy, 
psychology, sociology, and Spanish), the trairvees^ relative quantitative 
performance was uniformly imfjroved over their verbal performance. 

Rank ordering of the mean scores on the GRE-Q results in the list 
in Tabl^ 4.18. | 

Time Lapse Before Entry into Ti^aining Program 

/ 

Buswell, et.ai. (1966,yp. 8) found the number of years of teaching 
2xperience negatively relate;^ to research productivity, particularly after 
more than five years of exo/rience. He recognized that the factor of 
previous teadiing experience was interrelated with other factors, such as 



the age at time of the decision to go on for the doctorate and the age at 

/ 

the time of graduation J Presumably the factor of socialization into 
teaching and/or admini/stration career lines^(as opposed to a research 
career line) also becomes a stronger consideration as the years of teaching 
accumulate. Sieber and Lazarsfeld (1966, p. 273) concluded from their 
data that a requirement of previous professional experience for entry to 
research training actually reduced the product^ of researchers. 

Data are reported in Table 4.19 on the amount of time which lapsed 
between receipt of the most recent degree and entry to a Title IV graduate 
training program as developed in three studies: Fleury and Cappelluzzo, 
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Table 4.17 

Comparison of Mean Scores, of 1969-70 Title IV Trainees with Percentiles 
of First Year Graduate Students Scoring Lower on the Graduate 
Record Examination, Quantitative^ 



Training 
Programs 
Included 



Mean 

GRE-Q 

Score 



Percentile of First Year Graduate Students 
Scoring Lower on the Graduate Record 
Examination, Quantitative 



Biology . 


+^ 
CO 

B 
0) 

& 


Economics 


Engineering 


French 


Geology 


Government 


History 


Lit. in Eng 


Mathematics 


1 

Philosophy 


Physics 


Psychology 


Sociology 


Spanish 


All Fields ' 


73 


31 


48 


21 


86 


40 


79 


86 


86 


28 


58 


12 


63 


78 


98 


66 


72 


28 


44 


18 


85, 


36 


7b 


83 


84 


24 


55 


10 


61-^5 


98 


64 


71 


27 


42 


17 


84 


35 


75 


82 


83 


23 


54 


9 


60 


74 


'98 


63 


67 


27 


37 


13 


84 


33 


74 


80 


80 


16 


48 


5 


52 


72 


98 


61 



"Better" doctoral 
programs 

, Continuing doc/ 
toral prograDJS 

All doct/ral 
programs 

^Probationary & v 
Discontinued 
doctoral 
programs 

Subdoctoral 
programs 



631 
622 
616 

601 
521 



38 9 16 4 68 11 56 53 55 7 20 — 28 47 88 40 



Source: The Perfomance of First Year Graduate Students on the GRE , Educational 
. , , Testing Service, July, 1958- I 



See note to Table 4.15 for norm information- 
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Table CIS 

Ranking of 1969-70 Title IV Trainees arid Norm Groups in other Professions and 
the Disciplines on the Graduaiie Record Examination, Quantitative , 

/ ' 





Discipline, Profession, qt- Set 


Mean GRE-Q 


Standard 




Rank 


of Training Programs, 

// 


Score 


Deviation 


■N 


1 


Physics 


730 


67 ' 


233 


2 


Engineering / 


688 


83 


175 


3 


Mathematics 


681 


105 


289 


4 


Chemistry 


652 ' 


110 


280 


5 


Geology ^ 




98 


126 


6 


"Better" doctoral^^progi^g^ — ^ 


631 


97 


240 


7 


Economics ' -> 


624 


116 


260 


8 


Continuing doctoral programs 


622 


101 


300 


9 


All doctoral programs 


616 


102 


414 




Philosophy 


608 


117 


228 


11 


Probationary and discontinued doctoral 
programs 


601 


105 


114 


12 


Psychology ^ 


587 


ni 


258 


13 


All Fields " 


553 


137 


3812 


14 ' 


Biology y 


543 


123 


163 


15 


Subdoctoral Programs 


521 


86 


59 


16 


Government 


513 


129 


212 


17 


Sociology 


510 


142 


251 


18 


Literature in Engli sh 


504 


109 


^9 


19 


History 


502 


110 


20 


< 

French 


486 


105 


146 


21 


Spanish 


446 


77 


77 
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Sieber, and the present study. The assumption made in the latter that 
this was time spent in secuHng teaching experier^ce. Fleury and Cappelluzzo's 
population was composed of graduate trainees in Title IV training programs 
between September 1, 1966 and April, 1969. Thus, they were dealing with 
the same population as Sieber (the 1966-67 trainees) plus the, population 
of trainees who entered the program after 1966-67 but before April, 1969. 

In addition to determining the lapse of time for all graduate trainees, 
the Task Force staff analyzed separately the amount of time lapse for 
doctoral and subdoctoral trainees. These data are reported in the right 
hand portion of Table 4.19. The more interesting items in Table 4.19 
appear to be: 

(1) A greater proportion of trainees are entering their graduate 
programs within the five-year period Buswell found to be significant in 
the preparation of productive researchers: 86 percent in 1969-70 as com- 
pared to 72 or 73 percent in 1966-67. 

(2) The greatest change has occurred in the proportion who are 

s; 

beginning their advanced training immediately, or almost immediately, upon 
receipt of their degrees from 37 percent in Sieber 's population to 54 
percent of the 1969-70 trainees. 

(3) The increased proportion of trainees who are beginning ^their 
advanced training earlier may indicate a change in the recruitment practices 
of the doctoral programs. \ 

(4) The mean number of years elapsed prior to entry into advanced 
training declined by a full year from 1966-67 to 1969-70 (to 2.3 years) 
for all trainees, and-declined by fourteen^and one^half months (2.1 years) 
for the doctoral trainees. 
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There was some effort to recruit- into the subdoctoral programs 
Students who had just received their degrees,' but the modification was 
not sufficiently widespread to diminish very much the mean number of years- 
elapsed. 

Academic Background 

The trainees' academic background is an indication of the breadth 
of their preparation and the likelihood of their being well grounded in a 
substantive field other than education. Helss's {1966» p. 77) analysis 
of the academic background of 31 productive educational researchers led 
her to conclude that outstanding educational researchers will tend to 
show a background in a substantive rather than a professional field. 

Undergraduate major field . Buswell, et^aj,* (1966) gathered data 
on the undergraduate major field of 1954 and 1964 doctoral recipients in 
education which is con^arable to data assembled by the AERA Task Force on 
the 1969-70 Title IV doctoral trainees. These are presented in Table 4.20. 

The data in Table 4.20 indicate that: 

(1) Approximately three-fourths of the 1969-70 trainees earned an 
undergraduate degree outside of education. (This compares to 93 percent 
of Heiss's productive educational researchers.) 

(2) A higher proportion of the trainees in probationary and dis- 
continued doctoral programs held undergraduate degr*ees in education 

32 percent as compared to 20 percent of the trainees in those doctoral 
programs which were scheduled for continuation. 

(3) A further reduction in the nunfcer of doctoral programs being 
supported would likely produce such a small (1-2 percent) decrease in the 
proportion of trainees who had earned undergraduate degrees in education 
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that the "improvement" (if Jt be such) would not begin to offset the 
loss of 88 traineeships.. 

Graduate major field . Data on the graduate major fields of the 
1969-70 trainees appear in Table 4. 2K The only data available on 
trainees seeking the master's or another subdoctoral degree were for 
?hos^ in all Title IV programs. ^ / 

Notice shou]d be taken of the large number of trainees for whom 
no data were available (from 35 to 40 percent of the trainees in various 
combinations of doctoral programs and 68 percent of the sqbdoctoral 
trainees). A reconputation of the proportion of trainees in education 
and the other major fields with the number of trainees for which no 
data were available excluded substantially increases the proportion 
of trainees who had earned graduate degrees in^education, as indicated 
for the doctoral trainees in Table 4.22t 

Even in the "better" programs, the 46 percent figure for trainees 
with master's degrees in education does not compare favorably with 
the 33 percent of Heiss's productive researchers who had earned master's 
degrees in education. On the other hand, however, it does indicate 
that more than half of the trainees in the "better" programs vyere 
recruited into the Title IV doctoral training programs from fields 
outside of education^ 

In fact, the proportion of 1969-70 trainees recruited from 
outside education may be somewhat higher than that indicated in Table 
4,22. Some, at least, of the Title IV trainees entered the research 
training program with only the baccalaureate. If a person in this 
group happened to be part of a training program which required that he 
earp a master's degree ^ route to the doctorate, then he would appear 
in th^ category of those with education as a master's field. To an * 
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Table 4.22 

Graduate Major Fields of 1969-70 Doctoral Trainees 
for Whom Data Mere Available 



Graduate Major Field 



Sets of Doctoral Programs ^ 



Probationary . 

All Continuing n^crH^t^nnoH "Better" 
Programs Programs ^^Jf^J]^;;;^"^^ Programs 



Education 

Social and Behavioral 
Sciences 

Natural Sciences 

Humanities 

Other 

TOTAL 
N = 



53% 


52% 


56% 


46% 


.T 


" t 


18 


21 


16 


16 


16 


1^ • 


2 


2 


1 


2 


10 


10 


10 


12 


100% 


99% 


101% . 


100% 


(411) 


(278) 


(133) 


'I2I6) 



i 

unknown degree, the inclusion of such individuals tends to inflate the 
figures for trainees holding graduate degrees jn education. At, the 
same time, it tends to minimize the numbers of those who are regarded ■ 
as having been recruited to Title IV programs from outside the afield 

of education. * # 

\. ' ^ 

An interesting comparison may be -madV between the major fi el d^ 
of education- doctorate recipients (and trainees) at the undergraduate 
level and at the master's degree level. ^ Table 4.23 contains data 
drawn from Tables^ 4.2(J^ and 4.22 to make that comparison. (Note, 
however, that no master's degree data are available for 1954 and 1964 
graduates.) ^ ^ , ^ * 

• Table 4.23 



Major Fields of Earned Undergraduate and blaster's Level Degrees, 
1954 and 1964 Doctoral Recipients and 1969-70 Doctoral Trainees 





Undergraduate Degrees 


Master's Degrees 


Major Field 


1954 
Graduates 

- ^ ■ , 


Graduates 


1969-70 Doctoral 
Trainees, All 
Programs 


1969-70 Doctoral 
Trainees , "Al 1 
Programs 


Education 


•^23% 


30% 


26% 


IS 3/ij 


Social and 
Behavioral 
Sciences 


29 


. 23 


39 • 


19 


Natural 
Sciences 


20 


15 


23 


16 


Humanities 


17 


1? 


9 


2 

* i 


Other 


12 


13 


3 




TOTAL 


101% 


99% 


' 100% 


. 100% . 


H = . 


(818) . 


(1,750) 


(588)^ 


(411)' 



^Recomputed to exclude trainees for whom.no data were available. 
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The proportion of trainees who earned educati'on degrees'! doubled 
at the master^s 'level at the expanse of tell other major fields, but chiefly, 
at the expense, of the humanities and social and beh.avioral-'scilences. (It 
will be seen later in Table 4-/40 that an additional increase df about 25 per- 
cent occurs at the doctoral level.) As it was pointed out above, however, 
the proportion of master's degrees in education is. confounded to an unknown 
extent by the fact that "some trainees earn the master's en route to the 
doctorate as a part of the Title IV program. " - 

Grade Point Average 

Grade point ave^ag^s are^frequently accepted as indicators of' 
scholarship. It should be Jb^e in mind, however, that wh'f fe the GPA mSy 
be evidence of the stud^nt-^scholarship vis-a-vis * other stUdentsM'n the 
same location, ?it is-of only* the most general utility as one^moves away 

i 

from the particular department, school or university in/^hich the GPA was 

earned. For that reason, one should expect the scholarship-level trainees 

-to have high GPA's, and they do. Table 4,24 contains the mean GPA scores 

ft 

for the 1969-70 trainees. 

Table 4.24 

Mean Graduate Grade Point Averages for the 1969-70 Trainees 
• • ' Doctoral Programs Only 



ERJC 



Category ^ 


All Programs 


Continuing 
Programs 


Probationary 
*ahd Phased Out 
Programs 


"Better" 
Programs 




Mean 


s 


Mean • s 


Mean^ s 


Mean 


s 


Graduate ^GPA 


3.63 


.?6 


3.65 .25 


3.59 .27 


3.66 . 


26 




(414) 




(272) 


(142) 


(228) ■ 










Subdod 


toral Proqrams Only • 




Graduate GPA ' 


3.56 


.31 


3.68 .20 


3.55 !32 


.3.68 . 


20 


N / 


(104) 




(9) 

■ 


, '(95) 


"(9) 
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\ Other Indicators of Trainees' Potential Productivit y 

' / * ' and Corttributi on- 

. • \ 



A number of additional analyses relating tq various trainee 
characteristics were developed for- this study. On the surface these do 
not appear to be as strongly or directly related to potential fprodi>ction 
find/or' contribution ^s the five variables presented in the preceding 
section, but. the scholar in the*field will find'th'em of interest/ 



Personal iVariabTes 



ERIC 



Sex . There is a Hobson's Choice in the recrultnient of mailes or ^ 

females to the graduate training programs. Active researchers ate * 

- predominantly male, so it can be arc[ifea that only males shoul^Jbe' - 

^ recruited and awarded train^estrTps; to dos otherwise i^ to choose ^^e 

easy course of recruiting from among the large pool /of women in edoca'tion 

even though they are unlikely to follow research eareers* On the other 

♦ —'J 
hand, it is suggested that although women in education compose the most 

accessible recruitment pool for educational research, few women are re- 

c.ruited into educational research ajid have opportunities to go on to be 

productive resear*chers . Therefore, it can be argued that attention should 

f be devoted (a) to attracting the more intellectually talented among women 

into educational j^^es^arch training, and (b) ttf making careers in educa- ♦ 

tional 'V^esearch more attractive to women. The reader must interpret the 

data on the sex of the trainees according to, which of t^ese views he favors < 

Table 4.2& presents perqpntage of males among researchers-at-large, 1966-67 

trainees* and 1969-70 trainees-. 

Marital sta t us . Busvyell, et al- (1966, p. 51) determined t/iat 

83 percent of the 1964 'dqctorates in education were married at the tflne 

they receivejd the doctorate. Sieber (1968, pp* 82-3) states that, a . ^ 
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conservative estimate of the proportion of 1966-67 trainees who were 
married during their graduate studies was 62 percent. (There appears to 
be an err:or in the calculation since the number of cases cited 586 
traiflees with dependents out of 771 cases — is actually 76 percent of 
the total.) The data on the 1969-70 trainees are presented in Table 4.26. 

Two items appear to be of interes^t in Table 4.26; 

"(1)* The .proportion of doctoral trainees who were married may have 

d^lined a possible concomitant of the influx of younger trainees 

1 

, reported in the analyses of age upon entry to training. 

(2) A smaller proportion i>f th* subdoctoral trainees were married, 
yet the age-upon-entry analyses indicated the subdoctoral trainees were 
older, whicjp* appears to be an anomaly. A higher percentage of the sub- 
doctoral trainees were women, but that doesn't seem to be sufficient 
explanation for the anomaly. 

Number of dependents . At the time of their graduation, only 17 
■percent of BuswelTs population. of 1964 doctorates had no dependents. The 
average number for ^the entire group (of 1750) was 2.7 dependents; one-third 
had four or more dependents at the ti\e of graduation (Buswell, et al-, 
1966, p. 52). 5ieber (1968, p. 82' found the^^?iean number of dependents 

'for his entire- group (of 717) was 1.5 deperrdents, but his figure was for 

-students in tf)^ midst of their graduate 'training, ratBer than at the end 
of Jt. The- data for the entire group of 1969-70 trainees indicated that 

"the mean number of dependents had decreased from Sieber*s finding of * 
1.5 dependents to 1.36 dependent? 4^e "^able 4.27). 



\ 

\ 



/ 
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Table 4.25. 

Percentage of Males Among Researchers-At-Large, the 
1966-67 Trainees, and th^ 1969-70 Trainees 





* Researchers - 
at-Large^ 


1966-67 
Tratnees ^ 


1969-70 Trainees 




•All , ^ 
Trainees Doctoral Subdoctoral ' . » 


Male 
N 


86% 
(3,907) 


73% 
(774) 


68% 71% 56' • 
(797) (651 ) (( 146) 


^Source; 


Sieber, 1968, 


p. 41. ^ 






\^ Table 4.26 ^ 
Marital Status .of 1^969-70 Title IV Graduate Traijjj^^fes \ 



^ . Jlarital Status 


A^l Trainees in 
Al 1 P-rograrm 


Doctora^l Trainees 
in All Programs, 


^ — , 

Subdoctoral Trainees 
in All Programs^ \ 




N 


t 


N ■ %• 




I * 


Married 


539 


68 


4bf 


70 


84 


■ 58 , 


Single , 


222 


28 




> 25 


.. 59 


, 4(/ • 


Divorced or Widowed . 


27 


3 " ' 


25 




2 


1 


i/o Data ' • 


• 9 


* 1 






1 


1 


Total 


, 797 




lOU 




100*' 


146 


TOO 



Table 4.27 \ 



\ 



Mean Number of 'DeoendentS of 1969-70 Title IV Graduate Trainees 



1 



Category' 



r 



/ V . ' ^ Mean 

• • * 

Number of dependents K36 

^^.Ng-, * * ' .(781) ' 
- ' — ^ — — - 



All Trainees .in- ; Doctoral- Tra-rfiees Subdoctoral Jrainees 
All Programs, J in All ProqraHts Jn All Programs 

s 



15 



ERLC 



. Ne-da.ta 



0> 



(16) 



j Mean"' 's 
1.37 .15 


Mean 


1.30 


1 (537) *^ y- 


(144) • 


(14) , 





17 



\ 

\ 
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p Data from Tables 4,26 and 4.27 can be combined to give a clearer pictur 

j of the average number 6f dependents per married student, as follows: 

1969-70 Trainees 



Number of married 
trainees 



All Trainees in 
All Programs 

539 



Total number o> dependent? 1,060 

/7\^^rage number of 1*97 
dependents' peV* married 
trainee • - 



Subdoctoral 
Doctoral Trainees Trainees in 
in All Programs 



455 

873 
1.91 



\ 84 

187 
2.22 



These data-compare to an average, number of dependVats per married , 

8 ' 

trainee in 1966-67 of 2.66. 



Academic Background \* 

Degree(s) earned . The data on undergraduate and graduate degrees 
earjied b^f^the 1969-70 trainees are presented in Table 4.28. The reader should 
note that'no data were available on earned graduate degrees for 68 percent 
of the subdoctorgl trainees undoubtedly because most were ev^n then in 
the process of earning their first graduate degree. The size ofj the "no 
data" category is so larg^that little attention should be given the distri- 
bution of subdpctoral trainees across the various master's degrees. 

1 

The year^'s difference between the dopto,ral and subdoctoraT trainees 
in receipt of the de'gV-ee conforms precisely to the difference in time elapsed 



Table 44 in Sieber\l968) snows 430 trainees (56 percent of 767) 
with one or more dependents and a total number of dependents of 1J46, or 
an average of l2,667 dependents for each trainee who claimed dependents. 



( 
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between the trainees* receipt of their highest degree and entry into a 
Title IV training program (reported in Table 4.19). It also appears to reflect 
faithfully the difference in mean ages reported immediately following 
Table4J0 {28.5 years for doctoral ^ainees; .30.0 for subdoctoral trainees). 

Table 4.28 . \ 
Undergraduate and Graduate Degrees Earned by 1969-70 Tr^aihees 



/ 



Type of Degree 
Earned 


Doctoral 


Trainees 


Subdoctoral Trainees 


N 




% 


N 




Undergraduate 












B.A. 


350 




54 


63 


43 


B.S. 


. 209 




32 


71 


49 


B. Ed. 


' 29 




4 


11 


8 


Other 


16 




3 


1 


1 


No Data 






7 






Total 

> 


./ 651 




100 


'l46 


101 


Graduate 


V 










M.A. 


192 




30 


13 




M.S. / 


92 




14 1 


21 


14 


' M.Ed, tMAT) • ' 


98 




15 


9 


6 


Other 


28 




4 


4 


3 


No Data 


241 




37 


99 


68 


Total 


651 




100 


146 


100 


Date of receipt of degree{s). The 


mean 


date of 


\ 

receipt of their 


/ 

/ 


degrees was as follows for the 1969-70 


trai 


nees : 








Doctoral 


Trainees 


s 

Subdoctoral Trai 


pees 


Undergraduate Degree 
Graduate Degree 


May, 
June, 


1963 
1966 




May, 1962 
June, 1965 





c 

.^-j.-- — ..^ r 

'er|c 
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Quality of institutions which granted degrees . The extent to which 
the previous degrees earned by the 1969-70 trainees were granted by insti- 
tutions which had be,en cited by Keniston for the quality of their research 
was determined. These data, reported in Table 4.29, indicate that at least 
14 percent of the doctoral trainees earned their undergraduate degrees at 
institutions cited for the quality of their research, versus 8 percent ol(, 
the subdoctoral trainees. The doctoral trainees substantially improved 
that proportion (to 26 percent) at the graduate level. The large "no data" 
category for subdoctoral trainees confuses the results for that group. 



Table 4.29 



/ 



Appearance of Institutions Which Granted Degrees to 
1969-70 Trainees Among Institutions Cited by Keniston for the 
Quality of Their Research ' i 



Appearance Ainong 
Institutions Cited 
for Quality Research 



Appears among insti 
tutions citecLlojr 
research quality 

Does not appear 

rlo Data 

TOTAL 



Undergraduate Institution 



Doctoral 
Trainees 
Only 



N 



Subdoctoral 
Trainees 
Only 



N 




512 



50 



T4 !\ n 

I t 

I 

79 j 135 

i8 . 



i 651 101 146 



8 
92 

100 



Graduate Institution 



Doctoral 
Trainees 
Only 



Subdoctoral 
Trainees 
Only 



N 



% 



166 26 

378 58 
107 16 



70 
67 



48 
46 



651 100 . 146 100 
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Size of degree granting inst^utions . The size of the Jnsti tutions 
which granted undergraduate and graduate degrees to the 1969-70 trainees is 
reported in Table 4.30. As might be expected, the whole group of trainees 
moved toward larger institutions as they took up their graduate work. 

I 

Table 4.30 

. Size of Institutions in Which 1969-70 Trainees Earned 
\ Undergraduate and Graduate Degrees 



Size of Degree 
Granting Institution 


Undergraduate Institution 


Graduate Institution 


Doctoral 
Trainees 
Only 


Subdoctoral 
Trainees»\ 
Only \ 


Doctoral 
Trainees 
Only 


Subdoctoral 
i Trainees 
Only 




' N 


% 


N % ' 


N 


% i 


N 


% 


Under 4,000 


182 


28 


37 


25 


26 


4 


11 


8 


4,000-10,000 


112 


17 


46 


31 


106 


. 16 


32 


22 


10,000-20,000 


133 


20 


43 


30 


200 


31 


25 


17 


20,000-30,000 


95 


15 


8 


6 


96 


15 


5 


3 


Over 30,000 


68 


10 


10 


7 


107 


16 


6 


, 4 


<■ No data 


61 


9 


2 


1 


116 


18 


67 


46. 


TOTAL 


651 


99 


146 


100 


651 


100 


146 


100 



State in which deg ree instituti on located ^ The states in which the 
largest number of 1969-70 trainees earjied their undergraduate degrees were, 
for the most part, also the states in which the greaiost number of trainees 
eafrned their graduate degrees, as reported in Table 4.31, 

Not much can be said about this geographic distribution of trainees 
vis-a- vis trainees in general, other than that there is a definite correlation 

ERIC N ^ 
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Table 4.31 

States in Which Title IV Trainees' Degree -granting 
Institutions are Located 



States in Which Degree 

indtitUtlOnS arc 

Located 


li 


ndergraduate 


Degree 


Graduate Degree 


Rank 


No. of 
Trainees 


Percent of 
Trainees 


Rank 

r\ui In 


No. of 

1 1 U 1 1 ICC ^ 


Percent of 

1 1 U i I ICC d 


"California 


3 


62 


7.8 


2 


54 


6 8 


Connecticut 


11 


25 ' 


3.1 








Florida 


7 


32 


4.0 


7 


27 


3.4 


, Illinois 


5 


42 


5.3 


5 


35 


4.4 


Indiana 


12 


22 


2.8 


10 


20 


2.5 


Iowa 








11 


19 


2.4 


Massachusetts 


4 


53 


6.6 


2 


54 


6.8 


Michigan 


6 


38 


4.8 


5 


35 


4.4 


New York 


1 


121 


15.2 


1 


108 


13.6 




q 




J.J 


9 


22 


2.8 


Pennsyl vania 


2 


65 


8.2 


4 


37 


4.6 


Texas 


9 


26 


3.3 


8 


23 


2.9 


Wisconsin 


8 


27 


3.4 


11 


19 


2.4 


Others 




215 


27.2 




172 


22.0 


No Da td 




43 


6.4 




172 


22.0 


Total 




797 


100.4 




797 


101.0 



ERIC 
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between this distribution and the number of traineeships awarded by the 
Title IV Training Program, as indicated below. 

Table 4.32 

Ranking of Top States with Respect to Number of 
Traineeships, Undergraduate Degrees and Graduate Degrees 

Rank with Respect Rank in Number Rank in Number 
State to Number of of Undergraduate of Graduate 

Traineeships Degrees* Awarded Degrees Awarded 



New York 


1 


1 


1 


Massachusetts 


2 


4 


2 


Pennsylvania ^ 


3 


2 


4 


California j 


4 


3 


2 


Florida 


5 


7 


7 


Michigan 


6 


6 


5 


Texas 


7 


9 


V 8 


Wisconsin 


8 


8 


lY 


Iowa 


B 




11 


Illinois 


10 


5 


5 


Ohio 


10 


9 


9 



There was a subbtantial amount of interstate mobility (as reported 
below in Tajsie 4,33), but as the' rankings above irfdicate, the mobility 
apparently took place among the populous, university- rich states. 

\ 
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Table 4.33 



Number and Percent of 1969-70 Trainees Who Enrolled in a 
Title IV Training Program in Their State of Residence 



Is Program of Instruction 
in the Same State as the 
State of Residence? 


All 
Trainees 


Doctoral 
Trainees 


J UDQUi. i.ura 1 

Trainees 


N 


% 


N 


% 


N % 


Yes, in the same state. 


366 


56 


290 


51 


76 84 


No. 


292 


44 


277 


49' 


15 16 


TOTAL 


658 


100 


567 


100 


91 100 
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Before leaving the matter of interstate mobility, one further 
comparison may be drawn between the 1966-67 and 1969-70 trainees. As the 
data in Table 4.34 indicate, virtually no change occurred in the interstate 
mobility of trainees over the three-year period. 



Table 4,34 

Number and Percent of 1966-67 and 1969-70 Trainees Who 
Enrolled in a Title IV Training Program in the Same 
State as the State in Which Their Highest Degree Was Earned 



Is Program of Instruction 
in i:he Sarrie State as the 



1966-67. 
Trainees*^ 



1969-70 
Trainees 



Highest Degree Institution? 


N 


% 


N 


% 


Yes, in the same state. 


428 


58 


409 


59 


No. 


309 


42 


282 


41 


TOTAL 


737 


100 


691 


100 



Source: Sieber, 1968, p, 54. 
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Employment Background 

Sleber raised the question of whether the 1966-67 trainees* employ- 
ment experience Was such that it would have helped prepare the individual 
for a research career, or offered a predisposition in that direction. 
Results of a similar analysis for the 1969-70 trainees (presented in Table 
4.35 together with Sieber's results) suggest that somewhat more 1969-70 
trainees were recruited from among persons who had already shown a pre- 
dilection for research and were therefore more likely to remain in the 
educational research field after completing their degrees. 

Table 4,35 



Nature of Most Recent Employment of 1966-67 and 1969-70 
Trainees Prior to Their Entering a Title IV Training Program 





1966-67^ 
Trainees 






1969'- 


70 Trainees 




Category 


All 
Trainees 


Doctoral 
Only 


Subdoctoral 
Only 




N 


% 


N 


% 


N 


% 


N % 


In Education, Some 
Research 


54 


8 


112 


14 


105 


16 


7 


5 


In Education, No 
Research 


446 


66 


406 


51 


308 


47 


98 


67 


Not in Education, 
Some Research 


54 


8 


. 41 


5 


38 


6 


3 


2 


Not in Education, 
No Research 


122 


18 


160 


' 20 


131 


20 . 


29 


20 


Other or 
Indeterminate 






36 


5 


36 


6- 






No Data 






42 


5 


33 


5 


9 


6 


TOTAL 


676 


100 


797 


100 


651 


100 


146 


100 



'^Source: Sieber, 1968, p. 51. 
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In order to secure a somewhat better estimate of the magnitude or 
seriousness of tfie prior research employment experience, a separate analysis 
of the trainees^ most recent full-time employment was developed and 4s 
reported in Table 4.36, 



Table 4.36 ' 

Nature of Most Recent Full-Time Employment, 1969-70 Trainees 



Category 



Doctoral OnTy 



Subdoctoral Only 



N 



% 



In Education, Some Research 

In Education, No Research 

Not -in Education, Some 
Research 

Not in Education, No 
Research 

Other or Indeterminate 

No Data 

TOTAL 



54 8 
282 43 
30 5 



132 

15 

138 

651 



20 

2 
21 

99 



4. 
98 
3 

14 



27 



146 



3 

67 

2 

10 



19 



101 



It is clear from the data in Tables .4.35 and 4.36 that though a greater 
proportion of the trainees were recruited with some previous research 
experience, those trainees with a significant conr' lent to or employment 
in research were still a very small proportion of the total. 

As night be anticipated because of the younger age, fewer dependents, 
and greater likelihood of their having proceeded immediately from graduation 
to registration in a Title IV graining program, the doctoral trainees were 
less likely than the' subdoctor^al trainees to have worked full time at their 



most recent employment (Table 4.37). 

/ 
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Table 4.37 

Most Recent Employment of 1969-70 Trainees, Fu.ll- time or Part-time 



Type of Employment 


Doctoral Trainees Only - 


Su6doctoral Trainees Only 

* 


n 


% 


N ' % 


Full'tlme - 


402 


62 


112 


77 


Part-time 


188 


29 


20 




No Data 


61 


9 


14 » 


10 


. -TOTAL 


651 


100 


146 


101 



ERIC 
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The doctoral trainees were employed for a shorter period than the 
subdoctoral trainees, as reported below using mean data. 



Doctoral Trainees Subdoctoral Trainees 



Employment begun 
Employment ended 
Calendar length 



July, 1966 
June, 1968 
24 months 



July, 1965 
July, 1968 
37 months 



The mean employment period for all of the trainees was 25 months; 
by comparison, Sieber (1968, p* 49) reported a mean employment period for 
the 1966-67 trainees of 31.2 months (2.6 yeans). 

Professional Gogis 

Da ta"^ were obtained on selected professional goals of the 1969-70 
trainees. ' 

Type of» degree sought . When compared with the 1966 trainees, a \ 
smaller proportion of the 1969-70 doctoral trainees we^re seeking the Ed.D. * 
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degree (Table 4.3^). Inexplicably, since the' number of stibdoctoral and 
doctoral traineeships supported by Title IV remained essentially stable 
over the three-year period, the proportion of trainees seeking subdoctoral • 
degrees also declined; perhaps the differencetis traceable to the 60 cases 
for which no data werl^ obtained. The marked- increase in- the proportion 
of trainees seeking the Ph.D. — the "res^earch degree" -- is an encouraging 

sign. \ . ' . " 

Table 4.38 

Degree Sought by 1966-67 and 1969-70 Trainees 



1966-67 Trainees 



Degree Sought 



T969-70 Trainees 



N 



% 



Subdoctoral 
Ed.D. 
Ph.D. 
Others 

TOTAL 




19 
25 
54 

2 

100 



i07 
125 
505 

737 



15 
17. 
69 

101 



Source: Sieber, 1968, p. 57a, 



The data on degrees sought by the 1969-70 doctoral tradnees (Table 4.39) 
reveal an item of interest; tlie programs placed on probation or dis- 
continued featured a higher proportion of students seeking, the Ed.D. degree. 



There is an even greater bias 
in the "better" programs. It 



toward the Ph'.D. degree in the data on trainees 
pcems probable that the decision to elect/ 



the Ph.D. as the degree sought! is related to the pride felt by directors 



Table 4<39 

Degree Sought by 19j6§-70 J)octoral Trainees 



.Degree Sought 


All 
Programs 

// 


/ 

Ctintinuing . 
/Programs 


r 

Probationary , 
and. Phased Oi^ 
Prograits/, ■. 


/ . 1 

'Better" 
Programs 




W /' 


N " % 


N 


/%■ 


N 




Ed.D. 


/ 
122"^ 19 

504 ,77 


68 


15 


/ 54 


26 


39 


11 


. Ph.D. /. 

/ 

Other / 


363' 


81 /. 


141 


69 


305 - 


85 


/■17 3 


. 7 


I 2 


10 


5 


^6 


"Z . • 


/ 

No Data 


8 . 1 ' 


8 


,. 2 






8 


2 


* 

TOTAL 


651- 100 


446 


100 


205' 

* 


l/)0 . 


358 


100 



'» 
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and students in being part of ah outstanding research program; hence, 

" ' 9 • • 

they select the higher status, "research" degree --the Ph.D. 

Field of highest deg ree sought. Two presentations will be made 
on the field of the highest degree sought. The first presentation focuses 
on the fields of education, psychology and the other social sciences (Table 
4.40), Clearly, there was little movement in the fields being pursued 
by the trainees between 1966-57 and'^g-yo. 

Table 4.40 

7 

Education, Psychplogy, anS Other Social Sgience Concentrations in 
Fields of Hiahest Degree Sought by Researchers.-at-Large , and by 
the 1966-67 and 1969-70 Trainees 



^. ^ ^. ^ , Researchers-at- 1966-67^ ' 1969-70 
Field of Highest ^arge (1964) ^ Trainees^ Trainees 





N 


nf 


N 


h ' 


N 




Education (includ-' 
ing Ed. Psych) ' \ 


2,201 


57 


562 


11 


565 


11 


Psychology 


1 ,274 


33 


102 


14 


83 


11 


Other Social 
Sciences 


270 


7 , 


44 


6 


63 


8 


Others ,' 


116" 


3 


22 


3 


25 


4 


TOTAL \ 


3,861 


100 


730 


100 


736 


100 


^Source; Sieber, 


1968, p. 


73, 











Data on the relative correlations of Ph.D. and Ed.D., degrees with 
subsequent productivity in educational research are not conclusive; Lliore 
are studies supporting proponents of the Ph.D.. as the research degree and 
those that suggest that degree type makes no difference in later productivity. 
Although It is not possible to review the studies here, suffice it to say that 
anv trends across studies seem to favor the position that the Ph.D. is a more 
appr^aTe degree for neophyte researchers. Whether true or not, this opinion 
is held by many and has doubtlessly resulted in many strong research programs 
not offering the Fd.D. degree at all-. 
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For the 1969-70 trainees, it was also learned that (1) most of 
the "Other Social Sciences" category consisted of sociology -(six percent 
out of eight percent) and (2) the "Others" category consisted of humanities 
(two percent), natural sciences (one percent) and other (one percent). 

Data on the field of the highes^ degree sought by 1969-70 trainees 
alone are presented in Table 4.41. These figures reveal only minor differences 
among the continuing, the discontinued and the "better" doctoral programs. 
Again, the large "no data" category in the subdoctor^l section makes inter- 
pretation of these data difficult. 

Professional identification . Data were gathered on the jobs 'for 
which trainees saw themselves bei ng "prepared. The 1969-70 trainees 
identified professionally with the jobs listed i,i Table 4.42. 

The subtotals are inserted in Table 4.42 to facilitate comparison, of 
these jobs with a different set of job titles Sieber used to categorize 
the "Vocational identification" of the 1966-67 trainees. The job titles 
Sieber used, with the proportion of 1966-67 trainees so classified, are 
presented below in Table 4.43. Data for comparable groups of jobs, as 
indicated by the subtotals, are presented to the right for comparison. The 
reader will want to question whether the comparisons made are valid ones 
(for example, the juxtaposition of "student" and "undecided"). 

' To the extent that theVe is validity in the comparisons drawn between 
dissimilar job titles, it becotnes clear that the professional identification 
of the 1966-67 trainees was nearly identical to that of the 1969-70 trainees. . 
Perhaps this may be explained by the fact that the two classes of trainees 
were located in the same training programs — and thereby constrained in the 
breadth of their job choices by the objectives of the training programs. 
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Table 4.41 

Field of Highest Degree Sought by 1969-70 Trainees 





Doctoral Programs Only 


rlcIO OT nigncSt 

uegrse oougnu 


. All 
Programs 


Continuing 
Programs 


Probationary 
and Phased Out 
Programs 


"Better" 
Programs 




vi 


h 


N 


% 


N 


% 


N 


% ^ 


Education 


482 


74 


332 


74 


150 


73 




73 


P^vrhol OGV 




1 1 


■44 


10 


26 


13 


39 


11 


Sociology 


46 


7 


36 


f 


10 


5 


o 1 


Q 
J 


Other Social and 
Behavioral Sciences 


17- 


3 


10 


2 


7 


3 


10 


3 


Natural Sciences 


6 


1 


3 


1 


3 


2 


3 • 


1 


Humanities J 


12 


2 


6 


1 


6 


3 


0 


0 


Other 


7' 


1 


3 


1 


0 


0 


3 


1 • 


No Data 


11 


2 


12 


3 


3 


2 


11 


3 


Total 


651 


101 


446 


100 


'205 


101 


358 


101 








Subdoctoral Programs Only 






Education 


83 


57 


14 


100 


69 


52 


9 


100 


Psychology 


13 


9 


0 


0 


13 


10 


0 


0 


No Data 


50 


34 


' 0 


0 


50 


38 


0 


• 0 


Total 


146 


100 


14 


100 


132 


100 


9 


100 
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Table 4.42 

Vocational Identification of 1969-70 Title^U/ Trainees^ 



Job 

\ ' 


All Trainees 


Doctoral 


Trainees 


Subdoctoral Trainees 


N 


% 


^ N 


% 


N 


% 


Admi ni strator -OT researcn 


25 


3-.1 


'20 


3.1 


5 


3.4 


Adfnini strator ot evaiuaLion 


1 


.1 




J 


1 


.7 


AQimmpurator ot ucVciupniciiL 


2 


.3 


1 


.2 


1 


.7 


Administrator of diffusion 




^ _ 








— 


Other academic administration 


2 


.3 


2 


• 3 ■ 


-- 


— 
1.4 


Elem./Sec, administration 


13 


1.6 


1 1 


1 7 


2 


Other admim strati on 


4 


.5 


3 


.5 


1 


.7 


(Subtotal fij 


(47) 


(5.9) 


(37) 


(5.8) 


(10)^ 


(6.9) 


Jeacher of research 


12 


1.5 


n 


1.7 


1 


.7 


Teacher of evaluation 











-- 


— 


Teacher of development 




^ — ^ 






— 


— 


Teacher of diffusion 


__ 


_ _ _ 






— 


— 


Other university instruction 


36 


4.5 




5 5 






Elem./Sec, teacher 


17 


2.1 


Q 


1 .4 


8 




Teacher/researcher 


176 


^.1 


1 / i 


CD • 0 


5 


3.4 


Teacher/administrator 


8 


1.0 


6 


.9 


2 


1.4 


Other instructor 


13 


1 .6 


13 


2.0 




(11.0) 


( Subtotal #2) • 


(262) 


(32.8) 


(246) 


(37.8) . 


(16) 


Research Associate 


60. 


7.5 


48 


7.4 


12 


8.2 


Evaluation Associate 


7 


.9 


5 


.8 


2 


1.4 


Product developer 


4 


.5 


"4 


.6 






Diffusion specialist 


1 


.1 


1 


.2 


1 


.7 


Researcher/admin i strator 


7 


.9 


6 


.9 


( Subtotal #3) 


(79) 


(9.9) 


(64) 


(9.9) 


(15) 


(10.3) 



(Continued) 
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Table 4.42 (Continued) 
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All Trainees 


Doctoral 


Trainees 


Subdoctoral 


Trai nees 


N % 


N % 


N 


/Q 


298 37.4 


232 


35.6 


66 


45.2 


(298) (37.4) 


(232). 


(35.6) 


(66) ■ , 


(45.2) 


31 3.9 


21 


3.2 


10 


7.5 


80 10.0 


51 f 


7.8 


29 


19.2 


797 99.9 


651 - 


100.1 


146 


100.1 



Job 



Undecided 

( Subtotal #4) 

Current Status or Unknown- 
No Data 

Total 



^The professional identification data presented here suggest that the 
Title'IV programs are unlikely (as presently constituted) to produce arge 
numbers of educational development, evaluation, or diffusion personnel. 
Indeed, with few exceptions, the existing programs were not designed for 
that purpose. It should bejiot^. however, that the data reported in this 
paper probably underestimate t.o some unknown degree the number of Title iv 
trainees who actually take jobs in these research-related roles. An, 
inspection of appointment forms of trainees in two programs known to the 
writers showed that over 30 percent of the trainees in these two programs 
took jobs as developers or evaluators, even though, they had all given a 
professional identification of "researcher" on their appointment forms. 
This suggests that, an acquiescence set prompted by the respondent s Position 
as a trainee in an "educational research training program" may have color^ed 
the responses to some degree. However, this caution in interpreting the 
data does not negate the fact that the majority of persons in current 
training programs are preparing to be educational researchers and that 
training in research-related roles seems to receive only ancillary attention. 



Tabje 4.43. 

Vocational Identification of 1966-67 and 1969"-70 Title IV Trainees 



Sieber Job 
Designations 


1966-67 
Train, ees^ 


1969-70 
Trainees 


Task Force Job 
Designation^ 

< 




N ' 




tXeCU ui Yc 


45 


6 


47 


6.4 


Subtotal #1 


Teacher, Professor 


269 


36 


262 


36.5 


Subtotal #2 


Research Associate, 
Scienti^st, Statis- 
tician^ Other 


74 


'lO 


■-. 79 


11.0 


Subtotal #3 


Researcfi Assistant 


30 


4 








Student 


298 


40 


298 


41 .5 


Subtotal #4 


All Otner 


30 


4 


31 


3.9 




£ 

total 

. .__ -i — — 


•746 


100 


717 


99.3 





^Source: Sieber, 1968, p. 85 

I 

^RecomputGd to exclude 80 cases for which no data were available. 
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States/regions of planned employment. The states or regions in 
which the 1969-70 trainees planned to locate upon completion of their 
training are listed below, -The number of traineeships assigned to each 
of the five most popular states is also listed* so the effect of having 
traineeships located in a state can be estimated. 



i Table 4.44 

states and Regions of Planfied Employment for 1969-70 Title IV Trainees 



Site of Trainees' Traineeships Assigned 

Planned Residence N % of Total N % of Total 



New York 


52 


6.5 


145 


18.2 


California 


29 


3.6 


59 


7.4 


Massachusetts 


25 


3.1 


66 


8.3 


Pennsylvania 


•24 , 


3.0 


64 


8.0 


Florida 


17 


2.1 


52 


6.5 


East 


. 45 


5.6 






West 


29 


3.6 






Midwest . ■ 


28 


3.5 






Other Regions 


132 


16.8 






Undeci ded 


302 


37.9 






No data 


114 


14.3 






TOTAL 


797 


100.0 







The data in Table 4.44 Indicate that one- third to one-half of the 
trainees wish to remain in the state in which they are trained; this empha- 
sizes the desirability of maintaining some geographic spread in the 
training offered. The large number of undecided trainees plus the regional 
choices make it difficult to pursue this particular analysis further; a 
follow-up study of individuals is needed a few years hence to obtain 
better information on placement. 
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Observations and Conciuslons 

!• The 1969-70 trainees were found to be very much like the 1966-67 

trainees in many of the characteristics analyzed. These similarities 
probably result from the fact that both sets of trainees occupied 
'TraineQships in the same programs and were trained by essentially the 
same people. 

2. There were also found to be several noticeable differences between 
1966-&7 and 1969-70 trainees. The more interesting of these were 
the following: 

(a) The 1969-70 trainee^ were younger upon entry and will be younger 
upon graduation {from J to ,4 years) than the 1966-67 trainees; 

* they were much younger {5.5 to 8 years) than the 1964 graduates, 

in education. 

(b) The emphasis placed on younger trainees produced an increase in 
the proportion of students wno were recruited directly from their 
previous degree program — from 37 percent in 1966-67 to 54 percent 
in 1969-70. 

ic) The more recent trainees had a somewhat broader disciplinary-base 
than did education students in earlier years, particularly with 
respect to academic work in the social and behavioral sciences, 

(d) The 1969-70 trainees were more frequently recruited from positions 
which involvid some research activity; this indicated that they 
would be more likely to stay in the field once they graduate. 
However,* the proportion was still quite small (19 percent). 

(e) There was a greater, tendency among the 1969-70 trainees to seek 

the Ph.O, degree (69 percent vs^. 5' percent of the 1966-67 trainees); 
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as the trainees betfin to look more like arts and sciences students 
on the other variables (age, talent, etc.) it may be that they 
will look more and more like them in terms '^f the degree sought 
as well. 

Although prior intuitive criticisms of the .quality of the training 
programs and of the trainees may have had some validity when applied 
to individual cases, the data assembled here indicate they cannot be 
applied with validity to the entire trainiag program or group of trainees. 
It was found that: 

(1) For the most part, the training progr?ms were located at good . 
trainirfg institutions; approximately two-thirds of the doctoral 
prograins were at institutions e1ted for the quality of their 
rf?search. Of the remaining institutions not aitad for research 
quality fier S£, many are nonetheless noted for overall institu- 
tional quality. ^ 

(b) A strong, interdisciplinary emphasis did exist in the Tit]e IV 
training program^, suggesting that tne continuing call for more 
interdisciplinary training may be based on naivete about those 
progr^jns. Specifically, it was found that (1) -three-fourths of 
the 1969-70 doctoral trainees were recruited from undergraduate 
fields outside of education, (2) approximately half held the 
master* s degree in fields outside education, (3) at the doctoral 
level, a full one-fourth of the trainees were getting their degrees 
outside of education, and (4) forty percent of the programs provided 
some interdisciplinary training, including some in which the degree 
awarded was an education degree. An increasingly larger proportion 
of the trainees did move into education as th^y progressed from 
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, one degree to another, but this convergenc/"also characterized 
Heiss's (1966) outstanding scholars and is a natural consequence 
■ , of increased specialization. Her work tended 4;o verify that 
outs tiding educational researchers will •'show a background in 
a substantive rather than a professional field (Heiss, 1966, p. 77, 
italics added)," not that the degree or the burden of advanced work 
should or ipust be in a substantive field, ' 
(c) The 1969-70 trainees, as a' group, were very talented academically. - 
The GRE and MAT scores they received were on a par with or higher 
than the scorgs received by a majority of the students in virtually 
every professional and substantive field referenced. 

4, The remaining area of greatest ignorance- in studying Title IV training 
is the substance of the . dining programs themselves. The character- 
istics of ihe trainees, the training directors, and research- uni ts have 
been well /described and, in some cases, have been indepepdently verified 

through the presen| study and Sieber's (1968) early analyses. What 

° ■ ' 

happens to -the trainees whi-le they are in the traini ng , program -- tteat 
is, the processing phase is currently known only to the. trainees and 
directors in the various programs. , 

5. In the absence of more adequate data on the training programs (e.g., 
data on the content and practicum experiences provided within each 

' program, subsequent positions and research productivity of graduates 
'from eacn program) no final judgements can be drawn about the recent 
USOE decision to discontinue approximately one-third of the existing 
Title iV training programs. However, the evidence on trainees presented' 
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in these pages indicates that: »^ 

(a) The process of selecting Title IV training programs for continua- ] 
tion (and terminating or placing, others on probation) has proven 
effective in retaini^ig programs in which trainee talent is highest. 
The total GRE scores in Table 4,12, for example, show a 75-point 
difference between the mean scores of doctoral trainees in con- 

^ tinued programs and those of the trainees in probationary and ^ - 
discontinued programs. 

(b) EvetTin the 38 programs that were discontinued or placed on pro- 
bation, the overall level of talent was sufficiently high that 
the f)eld of educational research was not "saved from mediocrity" 
by the termination of federal suppprt for trainees in these programs. 

(c) Termination of all or an addi^tfonal segment of the Title IV training 
programs would disrupt the preparation of a cadre , of students in, 
educational research who could compete successfully with the best 
students in any and all of the professions and disciplines examine 
here. Mere importantly, such action would doubtlessly result in 

T 

\ the loss of a large number of these students from the field of 
educational research altogether. 

The Title IV Graduate Research Training Programs, particularly at the 

i 

doctoral level, appear likely to produce successful educational researchers 
and are deserving of strong support. ^ 
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ALTERNATIVE APPROACHES TO EDUCATIONAL RESEARCH TRAINING 




The final area of investigation in the 1969-70 Task Force 
project was the consideration of alternative approaches to educational 
research training. This effort took two di T.'hich are reported 

in this chapter. 

In the first section, information is presented from a survey of 
a small sample of professional organizations. The survey was undertaken 
in order to learn of inservice training approaches wnlch mignL nold pro-* 
mise for the training of educational researchers. 

The second section is a brief description of three training 
possibilities directly stimulated by activities of the Task horce. Tne 
third section of this cliapter is a detailed examination of-one of those 
possibilities:: the use of simulation techniques in educational research 
training. In particular the adaptability of this technique to the 
teaching of certain essential skills and knowledge is considered. 
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A Survey of Inservice Training P rogrdins 
S elected Professional Orgamza'tio ns' 



Introduction 

As part of the general focus of the AERA Task For^'e Project on the 
.Training of Educational Research Personnel, it was stated that "particular 
attention 'is to be directed to the needs for and methods of training for 
upgrading [educational research personnel], newly-developed methodology and 
technology, a'nd the overcoming of potential obsolescence amogg educational 
research personnel employed in a variety of jo^s and roles" (Gagne, T^69, 
p. 6).. An ultimate«aim of the Task Force is the development of an educa- 
" tional research training program which will accomplish the following:; 

(1) "upgrade the skills K)f researchers who are now poorly 

tra i ned ; 

(2) inaintain the high level of competence of researchers 
now entering the field; _ 

'(3) "neach new skills ma4e necessary by innovations in educa- 
tional techniques and products; and o 

(4) broaden the base of personnel engaged in. activities '.call ing 
for the application, of educational research skills. 

At present, the major continuing educational program for educational 
researchers consists" of the presesSions held just before the annual AERA . 
meeting/ The s^atdd purpose of the program is 

to train educational" researchers in fundamental research skills, 
e.g.* experimental design, statistical analysis, survey techniques.. 

The 'Research Training Presessions ,are intended to be instruc- 
tional or dissemi native of established research techniques as opposed 
to gen erative of new su'bstantive problems ... It is also not the 
purpoTe \ to disseminate innovations in education (e.g., team 
• teaching . . '.) which are not properly research skills and techniques 
common to a large class of research activities. 

Preference for. participation . . . will be given to researchers who 
Kold a doctorate. (Glass, 1968, pp. 3-4) 



ERIC 



Proposals for AERA ;.resessions are solicited through announcements 
in the Educational Researcher and professional journals and inviLitiwns to 
leading researchers. The courses to be offered are selected froiM the I'ro- 
posals subiint^ted to the Presessions Committee, and are announced in publi- 
cations in the behavioral sciences. Applicants are screened l^y the, ^ijha it -iMn 
of the session for which thev- apply. No tuition fees have beer>^charged for 
sessions up to and incli/ding 1970. (Due to reduction of Federal funding, 
tuition will be charged for the 1971 presessions • ) 

'\ 

The 1968 evaluation report (Glass, 1968) includes several interestinj 
facts. Eleven courses were offered at the conference presessions, eacii 
of five days' duration; over 700 participants were enrolled.. Analysis of 
data from these presessions shov^ed that 15 percent of the participants cdine 
from public school systans, 71 percent from colleges and universities.. 
Over 80 percent held an earned doctorate. The data also showed that a 
disproportionate number of enrollees came from the upper Midwest (the AERA 
Conference was held in Chicago that year), and .thus gave evidence to the 
belief that travel costs were a factor in presession attendance.^ These 
data indicate that there may be a need for expanding the training nrograi.i 
and offering it in other regions of the country. 

Several vehicles have been mentioned as possible elements in an 
expanded training program for educational researchers:. 

(1) Presession courses, of the sort which have been successfully 
conducted by AERA, in a still greater variety of substantive 
areas; / 

(2) Courses of longer duration, application to techniques and skills 
which require longer study; 

- (3) Summer institutes, conducted by outstandinq scholars witfi 
particular arPdS of expertncss; 
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(4) Conferences which would emphasize the generation and organiza- 
tion of new knowledge; 

(5) Workshops for the teachers of educational researchers, empha- 
sizing methods of teacning specific techniques or methods; ^ 

(6) Development of instructional materials and products for distri- 
bution to AERA members. 

One step in assessing the likely utility of these and other vehicles 
is to "discover, describe, and evaluate' the applicability of methods for 
upgrading training oi' professional personnel carried out by other professional 
and scientific organizations" (Gagne", 1969 , p. 6) . This section is 
directed to that task. 

Inservice Education Programs of 
Other Professional Orga ni zations 

In order to discover the types of in-service training employed else- 
where, a sample of professional organizations was selected for contact by 
members of the Task Force staff. General information available to those 
conducttng the investigation yielded a list of organizations which were 
thought to have adequate programs of in-service education for their members. 
From this list, a final selection was based on the pragmatic criteria of 
geographic accessibility for interview by a staff member and willingness of 
an officer of the organization to set aside time for an interview.. The 
resulting list of six^ Professional organizations, including the name of the 
person interviewed in oach case, is as, 'follows:, 

The American Association of JunioV' Colleges -- Washington, D.C. 
(Roger Yarrington) ^ 

The American AsscciUion of School Administrators,-- Washington, D.C 
(William Ellena) 

The American Chaucal. Society - Washington, D. C, (Moses Passer) 

\ 



ERIC 



/ 305 

The Aiiierican Psychological Association Washington, \). C. 
(Will iam Slmriions) 

The Association for Computing Machinery New York City, (James Adams) 

The National Society of Professional Engineers Washinqton, u. C. 
(Paul Robbins) 

It can be seen from this list that several disciplines were represented 
in the sample.^ 

The interviewee from each organization was interviewed during late 1969 
and early 1970 by a member of the Task Force staff. The objective of these 
interviews was to learn the following about each organization's in-service 
training program:. 

(1) intents of the program, 

(2) audiences addressed, 

(3} types of training vehicle eniployed, 

(4) manners of determining course content, 

(5) methods of developing course materials^ 

(6) cost factors and means of financing, 

(7) methods and results of evaluations, if any. 

What follows is a brief account^of the information gathered from each organi- 
zation. 

The American Associatjon of Junior Coll eges. 

While the American Association of Junior Colleges (AAJC) co-sponsors 
numerous seminars and workshops developed by other organizations, id while 
it actively publicizes programs of private industry, foundations and educa- 
tional institutions, these efforts do not appear to be directed toward the 
in-service education cf teachers and administrators in two-year colleges. \ 
Such programs are addressed, rather, to students or potential students in 
various phases of occupational education and to potential employers of the 
' graduates of two-year colleges. An exception is found in AAJC's publications 



in the field of health and inedical'technol ogy education. The National 
Health Council and AAJC, with the support of the W, K. Kellogg Foundation, 
have taken the initiative in providing substantial guidance for those ddiiiinis- 
trators who wish to develop educational programs in medical technology 
within their institutions. The AAJC also publishes for its members an 
in-service guide of summer courses and workshops in junior college teaching 
and administration; this guide lists, by, region, the sumer offerings of 
colleges and universities throughout the country. The AAJC does not, however, 
participate in the development of these courses, nor does it endorse them. 

The American Association for School Administrators . 

The American Association of School Administrators (AASA) has an exten- 
sive program of in-service training, aiiploying several different vehicles.; 
The general audience for the program is all practicing school administrators, 
but the specific audience may vary with the type of presentation (e.g.., by 
geographical location or by the type of position held by potential partici- 
pants).- Except for one type of vehicle, which is invitational, the program 
is open to all practicing administrators. 

Since the details of the programs vary considerably with the types 
of presentation, each training vehicle is discussed separately. 

The Annual Convjntion is a meeting open to all members of AASA (non- 
members may join at the time of the convention). Several large discussion ^ 
groups meet at the time of the convention, with discussion topics determined ' 
from a survey of 4,000 menbers of the organization. The meeting program is 
organized by the national office of AASA and is financed by conference 
registration fees. 

Reg ional Co nferences^ are held annuajly in the 12 regions of th'j 
Association, and are open to all administrators within the region.. The con- 
ferences arp generally throe-day programs of lectures, panels and informa- 
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tion-sharing sessions, with topics determined by those in the region. The 

conference staff is identified and supplied by the national organization. 

Prograni planning is done on a'regional level, but funding is by national 

membership dues and there is no cost to the individual participants. 

Circuit Rides are very informal meetings in v;hich a member of the 

national staff of AASA makes a one-day stop, usually in a rural area, to talk 

with a group of local administrators. The meetings are unstructured and 

« 

topics are determined as they progress. Expenses incurred by the staff 
member are a par.t of the normal operating budget of the national office and 
are financed through the membership dues of the .organization. 

The above activities are not continuing education programs ger s£. 
Two programs that are more directly relevant are tfie following, 

Thejjational Acadeny for School Executives is a relatively new program 
of AASA and may be described as a traveling academy of seminars, clinics 
and workshops lasting from five to 12 days. Topics for Academy sessions 
are determined from the recommendations of the educational staff of the 
national organization and from those of state Academy leaders. Sites arc 
chosen on the basis of accessibility by air and automobile, and suitability 
of physical facilities. Sessions are generally limited to about 50 partici- 
pants. This prografii is financed on a fee basis, with a tuitiotj charge of 
$180 for the five-day courses and from $225 to $280 for sessions lasting 
longer than five days. In addition, travel and room and board expenses are 
borne by the enrollee, \^f the session takes place outside of his home city.: 

There are two Seminar s for Pro fessors each year, meeting in different 
parts of the country. These are three-day, invitational meetings briiujinn 
together approximately 35 university professors of school administration and 
five top superintendents of schools selected from throughout the country. 
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Participdnts for the seminars are chosen on the basis of qeographical reore- 
sentation, the number of years remaining in the profession, time available 
and degree of commi tiaent. The seminar program, which is determined by the 
educational staff of the national office, consists of formal presentation of 
case studies, with each presentation followed by in-depth discussion. Provi- 
sion is also made 'for overall review of the problems and solutions considered 
at the meeting. No tuition is charged for the seminars and AASA reimburses 
each participant for half his travel and living expenses. The cost to the 
individual for the rasiaining half of his expenses averages $200 to $250. 

The American Chemical Society . 

Late in 1964, the Education Office of the American Chemical Society 
(ACS) published a report on continuing education for chemists; this followed 
a survey of ACS members on the use and effectiveness of continuing education 
techniques which were then available, and on the perceived need for and 
potential effectiveness of new techniques. The report included an account 
of programs of other scientific organizations. A major recommmendation of 
the investigators was that ACS take the initiative in continuing education 
by (1) iimiediately providing a listing of courses and lectures available 
through university Extensions, NSF and the like; and (2) developinq pilot 
programs emf^loying techniques not previously used. 

The first formal educational prog-am offered by ACS, shortly following 

' Ji 

the survey, consisted of courses (of a few days duration) presented at the 
• Society's annual meetings. It was found that the great majority of the 
■• participants were Ph.D.s. In order, to avoifi an apparent neglect of bachelor's 

degree holders--the "bench" chemists-'^CS inaugurated a program of traveling 

"short" courses.- Ultimately, the courses offered at the national ineetinq 

were dropped. * 
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The current education program of the Society consists of threQ types 
of training vehicles--the short courses, film courses and tape courses, 
The film and tape courses are extremely limited at present, however, and the 
overall program thus consists essentially of the short courses and their 
''package" course versions, which are described below. 

The priinary purpose of the ACS program is to combat technical obsoles- 
cence. by providing organized courses of study for the association's meiibers.: 
The target audience comprises graduates in chemistry (holders of degrees 
from the bachelor's dpgree through the doctorate) who are otherwise unable 
to keep abreast of technical innovations in their field. 'Once the desired 
background for students in a course is determined, participants are chosen 
on a self-selection basis. Enrollees in all phases of the ACS program come 
primarily from private industry. 

The ACS short courses are organized courses of study lasting from two 
to three days; the Society specifically avoids the kind of presentation 
which summarizes recent advances in a particular area of interest.. The 
courses are sponsored by local chapters of ACS and are held in some easily 
accessible meeting place, such as a hotel or convention center. The Education 
Office of the national organization has responsibility for all program develoD- 
ment and adrninistra tion. Suggestions for course subjects are solicited from • 
outstanding chemists and from previous registrants in ACS courses; course 
topics and instructors are then selected by the education staff. Once the 
instructor has been chosen, any decisions concerning format, additional staff, 
materials for study and the like are made by him. The usual format is a 
combination of lecture, discussion and laboratory demonstration. All written 
materials, including outlines, assignments and supplementary readings, are 
prepared by the staff of the course. The Education Office has responsibility 
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for the printing of the matlrials and for their distribution, if desired, 
to course registrants in advance. All expenses for the development, presen- 
tation and administration of the program of 50 short courses are undertaken 
by the national ACS organization. The costs are met by fees charged to the 
participants, with the fees averaging $bO to $60 per person'for a two-day 
course. 

In an effort to reduce the per person cost of the short courses and 
thus make them available to a larger audience of practicing chemists, the 
education staff conceived the idea of presenting a number of the short courses 
in "package" versions. The package version of a short course has the saiiie . 
content as the original presentation, and the staff and all materials are 
furnished by the ACS Education Office. The essential difference is that the^ 
course is sponsored and presented in-bouse by a private company (or perhaps 
two smaller companies in the same area) rather than by an ACS local affiliate. 
The sponsoring company undertakes all local arrangements for the presentation, 
including the meeting place, audio-visual facilities, publicity, registration 
and record-keeping. The administrative savings thus made possible to the 
national organization are passed on in the form of lower individual costs 
for the courses. It has been found that course expenses can be reduced by as 
much as 30 to 40 percent through the package offerings. 

Each new course developed for the ACS program is evaluated, at the time 
of its pilot presentation, by the staff of the Education Office. There- 
after, there is participant evaluation of each presentation of th>?, course, 
with feedback, through the Education Office, to the course staff. 

The film and tape courses mentioned above are recent additions co the 
ACS education program and have not yet been fully developed. To date, one 
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film course has been produced, in four parts with an accompanying chart hooK. 
The first tape course is a series of six interrelated lectures with textual 
material. These courses are available, through either rental or-T)urchase, 
to private industry, local ACS affiliates, and college and university orqani- 
zations.: Other film and tape courses are currently in preparation. 

The Afp erican Psycholog i cal Associati on. 

The. primary educational function of the American Psychological Associa- 
tion (APA), as a national organization, is in the area of accreditation of 
graduate degree programs in psychology. The parent organization has no formal 
or informal orogram of in-service education for its members, but„by publi- 



cizing available training funds it does encourage university departments to 
develop such programs. While some of the state associations hold workshops, 
'and some of the 29 divisions of APA are planning educational presentations 
for the APA annual ^meetings, these are not yet fully enough developed to con- 
stitute an in-service education program. 

The Associat ion for Com puting Machinery . 



The continuing education program of ^ the Association for Comouting 
Machinery (ACM) is jreatly similar to that of the American Chemical Society, 
which was discussed above. 

The general audience for the program consists of all those employed 
in tne use of electronic computers and related equipment. Most participants 
come from private industry and they represent all fields within the area of 
the computer arts and all personnel levels. Prerequisites for individual 
courses are specified in detail in the licerature describing the program. 
Potentidl participants determine the appropriateness of their owri oreparr^- . 
tion and are then selected by ACM on a first-^come, first-served basis. 
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The major purpose, of the program is to provide instruction in new skills 

and techniques required in a very rapidly changing industry. This • instruc- 

\ ' . ■ 

tion occurs m a series of one- or two-day traveling seminars covering a 

variety of topics. Subjects for the seminars are selected by the Professional 
Uevelopmonf Conmittee of ACM from several sources:. (1) a listing sent to ACM 
iT-.einbers , {2) titles of articles in journals, (3) topics of papers presented 
at meetings, and (4) questionnaires cbmpleted by participants i'n earlier 
seminars. 

'Once the need for a course on a particular subject has been established, 
the 'seminar is developed in one of two ways. The Professional Developi.ient 
Committee may select an instructor who is directly known to thein-, or they 
may solicit proposals for the course from several agencies. If the latter, 
then one of the proposals submitted will be -chosen by the Committee and the 
agency placed under contract for the planning and presyitatf'ion of the course. 
The major criteria in the selectiorr of a proposal are rel'evance of the content 
to the needs of the target audience, experience of , the proposed instructors 
(as practitioners as well as theoreticians), the appropriateness of the level 
of presentation, the cost to ACM and the availability of the instructors for 
additional presentations ofUhe course. 

All instructional materials are developed by the instructor or' by the 
agency selected for the course, but the national office of ACM handles print- 
ing and distribution. Average total costs for a two-day seminar accommoda- 
ting about 50 participants are approximately $1500. Financing is on a fee 
basis, with the fee paid- by the participant or his employer. 

Course evaluation takes place on two levels. The first session of a new 
course (and a dress rehearsal, if possible) are evaluated by, a team from the 
Professional Development Coniinttee. Subsequent sessions of the course art 
evaluated by the participants. 
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The National Soci ety o f P r Q^^e s s i o n a 1 E n g i nee r b . 

Four years ago, the Professional Engineers in Industry (PEI), a subgroup, 
of the National Society of Professional Engineers (M$PE), conducted ^ survey^ 
of it5 memters to determine the extent of their participation in various kinds 
of continuing education programs. The kinds of study Tncludedr (1) college 
and university programs, (2) programs sponsored by private industry, and 
(3-) those presented by other professional organizations. PEI concluded fror; 
this^ investigation that private industry should take>pr imary responsibility 
for the continuing education of engineers by making twie available to enioloy- 
ees, organizing and conducting courses, defraying costs for education programs, 
and providing einployee motiva^tion through pay increases, promotions, etc. 

^ As a possible result of PEPs conclusions, NSPE has adopted a relativejy 
passive rolejn providing in-service education for engineers. The Society does 
present seminars at its national meeting, but these deaf with management pro- 
blems and not with engineering topics £er s£. In addition, some local chapters 
of NSPE sponsor refresher courses for those seeking engineering licences from 
the state. These are lecture courses staffed by faculty from local institu- 
tions and tuition is charged to cover expenses. 

The vehicle which appears to be most directly related to engineering 

, education is a series of programmed long courses which the .Society makes 
availabJe a>t cost to its mefnbers. These are packaged, self-instructional 
courses which are purchased in quantity 'from a company specializing in educa- 
tional products; the per course cost ranges from $3 to $50 and averages about 
$10. The course "enrol leeV pays the fee to NSPE, but the Society itself ^ 
in np way involved in the development of the educational material. ^ 
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Summary 

From the program descriptions just completed, it is evident that oven a 
small sample of professional organizations provides a wide variety of toth- 
niques for in-service training. Among the formal programs, however^-tliose 
developed by^the American Association of School Administrators, the Ainer-can 
Chemical Society, and the Association for Computing Machinery--there drer 
several parameters which can be examined in order to give a brief sketch) of 
the vehicles employed. These identifying features are shown in Table b)l fo 
AASA, ACS, and ACM, and comparable information is-fliven for the ALRA p/e- - 
sessions. . . ' 

• Although it is impossibly to give a single description which will be ' 
valid for each individual case, certain cofmiorT charatteristics may noted 
ia'the non-AERA programs; 

1. In general , a traveling coarse lasting from one to three days 

has been adopted— or has ^evolved— as an effective training 

vef>icle. 

2., In general , topics for presentation as courses are solici.ted 

informally from the membership or are selected from current # 

literature in the field. 
3^ In general , the Actual course content and presentation are 

developed by those within the parent organization, 
4< In general , participants are selected on a self -screening basis; 

i e., the applicant himself determines whether h£ has the desired 

background for the course. 
5. In general , a tuition fee is charged for the course; thu program 

is thereby made self-supporting 



In genial, it possible for most of the potential audience 
'*to participate in some part of the program without incurring 
significant ^ses for travel and lodging. (Whfle it is most 
unlikely that the entire program of 50 ACS courses will be 
pr^ented in Chicago, for example, during oqe year, it is fairly 
certairr that a chemist in that city will be able to choose froiii 
a number of ACS offerings without leavin) the Chicago area.) 
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Table 5.1 



Salient Features of Selected In-Service Training Programs 



Professional Organization 
and Training \/^hicle 



AASA 

Regional Conference 

National Academy for 
School Executives 

Seminars for Profs. 
ACS 

Short Courses and 
Package Courses 

ACM 

Traveling Seminars 

AERA 
Presessions ' 




^ This cateuory represents additional expePites-e.g . , for travel and housing-- 
which would be incurred by large numbers of potential participants. 

^ This cateacrv represents formal solicitation of tne organization's menibership 

0^ of p^e oJs participants in the education program. In all cases e topics 
' ■ offered were actually selected by the orga'ni zation, not by the membership- 

^ Although the regional conferences are stationary," there are 12 of them held 
annually in 12 separate regions of AASA. 



(Continued)' 
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Table 5>1 (Continue d) 



^ .Some sessions which are held in resort" areas rather than major population 
centers would^ require additional expenses. 

e 

Each participant pays half his expenses, ^ 
^ Tcr^tio'^ for package courses is less than for short courses. 

^ AERA solicits from its membership proposals to conduct courses on any topic, 
^ The instrructor,- the top'ic and the course outline are thus accepted or 
rejected as a package. In the other asso'ciations represented here, an 
instructor is designated by the organTzation to develop a course on a topic 
which has al^^eady been determined* 

h . \ • ' 

Members are asked to submit proposals for sessions, including specification 
of content, but the membership ^ not polled for perceptions of most needed 
topics. , J 
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Recon^nendations 



. The saiijple of proressij^ial orydnizations discussed above ib 
evidently quite small and is not representative of all professional associa- 
tions. In addition, iL should be noted that the kinds of knowledge required 
Dy a practitioner o^ one of the j^hysica'i sciences, for example and the kinds 
of skills which must be mastered by him, are quite likely to be very different 
from those required for the preparation of an educational researcher. One 
may state with some assurance, therefore, that the specific training techf 
niques which have been used successfully by one or another of these organiza- 
tions may not easily be adopted en bloc for the educational research community 
With these limitations and disclaimers in mind, however, it ib suggested 
' that certain elements of the training programs described above hold consi- 
derable promise for the improvement oT in-service training in educational 
research. On this basis, the following recommendations are presented for Task 
Force consideration. 

1. Any in-service training program in educational research must actively 
and imaginatively seek participants ^rom outside the membership of 
AERA. There are indications that considerable numbers of personnel 
who are currently involved in educational re<;earch or research-related 
activities are not associated with AERA (Worthen and Roaden, 1969,. 
' pp. 12-14, 29)., in fact, these persons may not even be readily iden- 
tifiable as educational researchers, 'since they "may in many instances 
function in a capacity involving educational research activities without 
necessarily having this function reflected in their job titles" (Gaqne, 
1969, p, 3), 
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2. A survey should be cunducted of the menbership of AERA, and other 
educational research and research-related personnel who can be iden- 
tified, in order to determine"* both the subj_ec_ts which need to be 
included in the education program and the training vehicles (e.g., 
presesSions, two- or three-day courses, summer institutes) by ;;hich 
they may be presented most effectively, 

3. The program should be constructed in such a: way that the non-university 
practitioners of educational research will be served. That is, courses 
should be offered which will provide skills specifically needed by 
those engaged in applied research, development, diffusion, and evalua-^ 
ticn. 

4. The program must be made available on a local level. It is nut suffi- 
cient to offer courses at the annual meeting of AERA, since the time 
and financial commitment required for attendance at such conventions 
severely limits the audience, 

5. A tuition fee should be charged. ,At present, there is no charge for 
zhe pr^esebsions. Unfortunately, however, the incentive provided by^ 
having tuition-free courses is offset by the expenses of travel and 
lodging at the presession site. If the program Is to be offered on 

a local or traveling basis, it will cost' more than it does now. »y 
charging tui tion, ^AERA can make tho program self-supporting and, at 
the same time, make it accessible to those who are now effectively 
excluded from participation J ^ 



It has been argued that although chanists, engineers, and others 
usually employed by private industry must avoid obsolescence or {jeopardize 
their jobs, no such pressures exist for most educational researchers. 
Whether or not educational researchers would be as willing to pay tuition 
costs for post-doctoral- training as persorjs in these other fields is a 
moot point that could be determined by testing this proposal. 



AERA might consider establishing a regional administrative structure 
for such training programs. Such a structure would almost certainly 
ajd in the implementation of recommendations 2 and 4. It is also 
likely that a regional AERA organization will seem less formidable, 
and therefore more accessible, to nnn-university personnel involved 
in applied research and research-related activities. The structure 
developed need not be elaborate and could, in fact, be quite informal. 
AERA should investigate the feasibility of sponsoring the production 
of instructional packages which could be made available to its manbers 



AERA should investigate the feasibility of compiling a list of avail- 
able fellowships, institutes, workshops, etc., in educational researcn 
and related activities and disseminating this list to its members. 
The Task Force should decide whether information of the type contained 



extensive survey of training practices of a' broader sample of profes- 
sional associations than the convenience sample used in the present 
survey. 



at a reasonable cost. 



herein. is sufficiently useful to warrant 
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Research Train ing Activities S timulated by The Task Force 

One of the objectives for 1969-70 Task Forpe operation was "To conduct 
a tryout and evaluation of at least two promising modes of training 
[research personnel]." Almost 12 percent of the fu^ds from the USOE gr^nt 

to the Task Force were designated for the implementation and evaluation of 

\ 

the two training modes. However, it became apparent during 1969-70 that 
information about certain characteristics of present trainees and graduates 
of ESEA Title IV graduate research training iprograms was essential to 
attain other objectives the Task Force had espoused. Since the original 
proposal contained no provision or funds for collecting and analyzing 
such data, the possibility of requesting additional funds for these activities 
was explored with the project officer in the Research Training Branch of the 
USOE NationaTcenter for Educational Research and Development. The result of 
these discussions was a joint decision of the project officer and the Task 
Force that the funds originally designated "for the tryout and evaluation of 
two modes of training should be used instead to collect and analyze the data 
on Title IV trainees that are reported in Chapter 4" of this report. 

Despite the decision to eliminate the "tryout and evaluation" 
objective from the formal objectives of the project, the Task Force continued 
to consider alternative training modes that seem to have potential for training 
educational researchers and research-related persd'nnel . Many alternative 
training techniques were discussed by the Task Force, and these deliberations 
stimulated efforts to design, implement and evaluate, some modes of training ^ 
that might be useful in preparing educational research personnel. Three such 
activities are described belovv* 
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Cassette Tapes on Research Topics 

One suggestion for training that was generated within the Task 
Force was the developiiient of a library of cassette tapes on research 
topics for use by researchers in keeping abreast of new developments 
in research methodology. The intent was to develop materials that could 
be used during "dead" time (e.g., conmuting, airplane travel, lunch) to 
learn more about important new research topics. The topics themselves 
.were of less concern to the Task Force than was testing whether or not 
available cassette tapes of this type would be used by researchers. 

The responsibility for development of these training mate; ials 
was given to W. J. Popham, with whom the idea originated. A. proposal 
to the USOE for development of a cassette tape library has been funded 
and is currently being implemented under the auspices of the Task Force- 
(The body of the proposal may be found in Appendix M.) As part of that 
project, an extensive evaluation of the effectiveness of this training 
technique will be conducted, resulting in decisions about its future 
utility for training research personnel in education. 

P>;;esessj_on_ Materia1s Development , ■ ■ 

Another suggestion that resulted from Task Force discussions was 
that transportable packages of training materials should be developed 
on several r'^search topics for which good curriculum materials are not 
now available. Rather than beginning such developmental efforts de novo, 
it was suggested that materials already developed for use ^n prior A£RA 
presessions for research training could serve as a basis for further 
materials development. Consequently, all prior AE,RA^ presessions (and 
postsessions) were sc?eened on the basis of two criteria: (a) is there 
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evidence that the content of the presession is of interest to a sufficient 
number of researchers and research trainers to warrant its developn^ent 
into an instructional package and (b) are the basic materials that are 
already developed of sufficient quality to serve as a basis for furtrier 
developmental etforts. 

Twelve presessions met both criteria and the directors of these 
presessions were encouraged by the Task Force to submit proposals for 
materials development grants under RFP 70-27 issued by the USOE Research 
Training Branch. Five presession directors agreed to submit such proposals 

r 

and, if funded, to develop packaged curriculum materials and submit tnem 

to rigorous evaluation. Two directors ultimately submitted proposals 

under RFP 70-27; one was accepted and will result in tne development of 

three course content modules to train research and research-related 

personnel to appraise research critically. The second proposal was 

rejected with suggestions for specific revisions and the recommendation 

that it be re-submitted in the later conipetition for materials development grants 

Simulate d Re search Activities 

In a recent issue of the £ducational_ Researcher (Vol. XX, No. 8, 
1969), the Task Force solicited the AERA membership ideas aoout new 
arid* effective means for training educational research and research-related 
personnel with the intent of assisting in finding funding sources for 
those ideas that seemed worthy of implementation. Thirty-three proposals 
were submitted by AERA members and were screened by use of the following 
criteria: 

A. General 

1. Each proposal should have definite potential for developing, 
maintaining or upgrading relevant research and research-related 
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skills through either pre- or in-service training. 

2. Each proposal should either:' (a) be designed to present 
content relating to the most needed skills and/or functionb, 
or (b) utilize a training technique that promibos to be 
generalizable across content areas. 



Speci fic 

!• Target Audience 

a. what type of person will be trained 

b. ^ number of trainees involved 

c. anticipated availability of trainees 

1. Director and Tentative Staff 

a, experience and capability 

b. degree of commitment of tentative staff 

3. Content 

a. clarity about content to be included 

b. importance of topic (need) 

c. usefulness of competencies to be developed 

4. Instructional Techniques 

a. appropriateness for content being presented 

b, organization of topics 

5. Proposed evaluation activities 

a. extent of ^ planning 

b. appropriateness uf evaluation design 

c. comprehensi veness 

6. Feasibility (adequacy of resources to conduct the program) 

a. cost (economic efficiency) 

b. facilities 

c. institutional support 

7. Coordination with other programs in progress to avoid 
duplication 

8. Potential for program continuation and expansion after 
funding period 

9. Breadth of probable impact, in tenns of research and researcn- 
related functions in varying institutional settings. 
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Only one of the suggested training techniques met the criteria 
.. • 2 
well enough to warrant further consideration. This was a proposal for 

investigating the possible application of simulation techniques to the 

3 

.,trc/ining of educational researchers. That investigation is reported in 

i 

the following section. 



Three other proposals met all criteria except that of economic 
feasibility. They wer^ designs for training centers not unlike innovative 
Title IV graduate research training programs, with costs ranging between 
$50,000 and $190,000 annually.- No funds at that level were available. 



The proposal was -submitted by Dr. William L. Goodwin, USOE post- 
doctoral research fellow at Harvard University. 
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Considerations in Developing Si mulation Materia l s for 

4 5 

Trainingon Educational Research ' 

To what extent is simulation an appropriate and potentially useful 
technique for the training 'of research and research-related personnel in 
education? How can it best be used? In what forui? In what topical areas? 
The purpose of this section is to provide partial answers to the questions 
above. No attempt, is made here to provide definitive specifications for 
developing simulation materials; only general guidelines are presented for 
Task Force consideration. 

The remainder of this section is divided into four major parts. 
In the first part, attention will be given .to the use of simulation for 
instructional purposes, in general, and its use for training personnel 
in educational roles, in particular. Other parts of this section will include 
consideration of (a) suggested steps in designing a simulation, (b) areas 
in research, development, diffusion, and evaluation most amenable to a simula- 
tion format and (c) issues pertinent to the tole of simulation In the area 
of research training, ' 



4 

This section originally appeared as Technical Paper No. 17 in the AERA 
Task Force series, under the autnorship of W.L. Goodwin and B.R. Wortnen. Only 
minor stylistic changes have been made for its presentation here. 

. 5 

The authors express appreciation to Donald R. Cruickshank and 
Paul A. Twelker for their advice and assistance in clarifying many concepts 
in this paper and for providing and guiding us to many useful references. Despite 
• the contribution of Drs. Cruickshank and Twelker to this paper, the responsibility 
for any misunderstandings or inadequacies in its content rests solely with tne 
authors. 
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In structio nal Uses of S imulation 

This portion is to familiarize the reader new to the area of simulation 
wHn present uses of the technique in a variety of insiructional programs.. 
Two excellent and complementary .reviews served extensively in providing 
material for this part (Cruickshank and Broadbent, 1970; and Twelker; 1969) 
and the reader ^seeking more information fs directed to them and to a recent 
ERIC basic reference paper by Twelker (1970a); these sources are useful in 
reviewing simulation's use in research, operational analysis, design and ^ 
development as welTas in Biilitary, government, medical and industrial 
SQ.ttings on topics no.fe considered here. 

For the purposes of this section, simulation is defined as "a 
representation of several variables in the same arrangement as they occur 
in a particular natural or arti ficiaT system" (Crawford, 196^), while tne 
term "simulation games," following Cruickshank and Broadbent (1970), ib used 
to denote simulations that involve competitive interactians between or among 
participants. It should be noted that these definitions have led to the 
develop^ment of many simulation systems and games that meet these definitional 
criteria to a greater or lesser extent., ^ * / 

Outside the area of training educators for various roles, simulation . . . 
t ' j ■ 

has made its most notable impact upon the instruction of students, nonnally 

in the form of simulation games. To date, the use of games as an 'instructional 

technique aimed at imparting subject content to pupils is intuitive. There 

is no solj^^vidence that it is more effective or efficient than other niodes 

(Cherryholmes, 1966; Garvey and Sei ler,* 1966 ; Inbar, 1966; and Twelker, 1969); 

several studies reported in Boocock and Schild (1968) and Robinson (1966) provide 

6 

additianal data on games' outcomes, but results are far from definitive. 

_ / 



Q The reader is directed to an exhaustive review of resean 

IERIC Fletcher (1969), 
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Rather, the use of simulation games is justified on tt^-basis. of their. 
being valuable for imparting decision-making skills (Dt:maree, 19ol, Parker 
and Downs, 1961; Twelker, 1967;- Western Behavioral Science Institute, 196b) 
as in management games (Cohen and Rehnman, 1961; Dill, 1961; Fulnn^^r, 1963) or, 
relatedly, for training students to select between alternate strategies (ADt,_ 
1966; Boocock, 1966; Schild, 1966)-. Abt (1966) suggests furti.er that .tUderit 
learning of processes simulated^ by the game, takes place, out no, evidence ;is 
presented for this. The belief that simulation games serve as motivational 
stimulants is widely expressed (Abt, 1966; Bruner, 1966; Cherryholmes , 1966; 
Coleman, 1960, 1966; and Sprague and Shirts, 1966). In short, claims for the 
utilizing of simulation are many, but primarily not based on research resuTts- 

Many different types of simulation are in 'common use as instructional 
'devi^,es. Some of the most interesting are described below. 

One simulation with the unique characteriki c of continuing sessions, 
for an hour or so daily throughout the errtire school year, warrants special^ .. 
note. Micro-society has been developed undeV the auspices of the Portola 
Institute (Dcbbs, et al_., 1968). In Mi-cro-society, upper elementary students 
, organise their classroom as a miniature sociaV system.. Each student has a 
rone, or multiple roles, and via the resultant interaction, observes and 
engages in activities necessary for maintenance of the system (e.g., book 
delivery s^vice; message delivery system; the selling of nnd-morning 
sustenance, office supplies, etc.; and court settlement of civil disputes): 

When the concept of student is expanded to include the television- 
viewing puDlic, the motivational qualities of simulation are again noted as . 
' during "The Most Dangerous Game" broadcast (on the 1950 Korean Crisis) that 
permitted home-viewer interaction (Lee, 1967, 1968), "Cabinets in Crisis" 
(on the 1950 Yugoslavia Crisis with Russia; reported in Twelker, 1969, p., I-3b) 
' and the National Driving Test administered over TV in the late 196.0's. ^ 
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Instructional uses of simulation have also appeared, or been sg.icjqested, 
in the general area of vocational education. Boocock's Life Career Game is^ 
designed t^give students a preview of the multiple ramifications of vocational 
choi^. Hamreus '(1969) has proposed instructional uses of simulation in 
di verse ^eas, such as auto repair, secretarial work, health education, consumer 
marketing, sheet metal fabrication, home economics, drafting, selling and 
merchandising, accounting and the likew 

A wide range of actual simulation materials has been developed for 
medical stujJents and is utilized for fioth teaching and assessment purposes. 
(Note ]a*ter in this section Schalock's caiiments on the use of simulation for 
assessments^) At the "paper and pencil" end of the continuum, a training exam 
involves sequential decisions on the part of the trainee-as he makes a 
diaqnosis based on case material he reads, erases an opaque material on the 
answer sheet to indicate jiis choice, and proceeds as directed hy Jthe a^rea 
of the answer sneet thus exposed* This continues through a series of stefh 
(note that this is a "oranching" technique with trainees often taking different 
routes) with feedback the student on the probabfe outcome of his procedural 
technique (Crawford and Lewy, 1965; McGuire and Babbott, 1967; Univeristy of 
m,inois, 1967a, b) ; uniquely, several paths may lead to a favorable putcome. 
At a somewhat different end of the continuum* an ingenious flesh-colored, 
skin-fitting suit, has been developed on which war wounds may be simulated and 
trainees judged on their efforts to treat same (Wooley and Audet, 1956). More 
recently, an. experisi ve compute r-contro i ' J patient has been developed for 
medical training; titled Sim One, it is highly realistic (Abrahamson, et_ i[. . 
1969). Material* aTso exist for role 'playing diagnostic and treatment inter- 
views. (Levine and McGuireo- 1968). In another inUrument, tape recordings of 



/ 

three case studies are interrupted ueriodicdlly to allow students to wr^te 

ten questions they would like to ask ttie pdtiont at that point; \u addition 

to a ski-11 score, attitudinal concern for the patient is also derived 

(Frederlksen , 1962d) , 

Most instructional uses of stimulation and simulation games have ^ 

involved materials of the paper-and-^^enci 1 variety, presumably to keep costs 

reasonable. It-is difficult, for example, to think of simulation devices 

used in public schools that are similar in complexity or cost to tne link 

trainer .developed to train pilot- he gyrating decks and "related equipnvint 

designed to train submarine connaaders , the -ingenious and extensive 

simulation vehicles and situatii)ns established in oonjunction with the space 

program,, Some fairny elaborate simulations have been introduced in tne scnools, 

however, such as the driver training package in which t^ie student operates 

basic dutomobile-like controls in response to stimuli presented via Vj^rious ^ 

projectors integrated through an electronic control system. 

% 

Simulation has been suggested as amenable to a second educational area^ 
separate from but related to instruction. Schalock (1969) proposes that simujla- 
tion^mi^'ght play a valuable role in effective measurement. \Essentially , via ! 
priQr work (Schalock, et al. , 1965; and Schalock and Beaird, 1968), it is 
sjggesled that simulation allows the creation of 'more realistic test situations 
(i.e., with Higher fidelity) permitting the elicitation of more complex responses. 
R^'sponses thus observed and judged are considered by Schalock to have greater 
predlcti^'e validity than more conmon [heasures derived from conventional examining 
means. Simulations, it is suggested, allow assessment of how on'e behaves in a 
If felike -si tuhtion, not merely ascertainment af how he tiiinks he should oenave, 
'Many of the uses of simulation for training personnel in educational roleo nave 
been developed along the lines implied by Schalock, but with emphasis on tne 
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realistic training' thus provided rather than on cons.i derations related 

"S. 

primarily to measurement, J 
\ 

Simulation in Training Personnel for Educational Roles . 

In a very real sense, the specific use of simulation for training 
educators has had a short history, having probably been initiated with the in>. 
basket test for training school administrators (Frederiksen, 'et a^. , 19b7; 
Frederiksen, 1962d; and Hemphill, et al, , 1962). The trainee primarily reads _J 
background material that sets the situation although some fi^lms and tapes are 
also used. For two days, the trainee plays the role of a school principal 
firstj coming on the job a few days before school opens. Via his in-basket, he 
receives and must make decisions on a series of conmuni cations (letters, 
messages, memos, etc.) typical of those that would confront a principal oefore 
the opening of school. Based on his decisions, a profile is prepared that in- 
dicates how his responses compared with others along the dimensions of analyzing 
situations, maintaining relationships, preparing information,, responding to out- 
siders, complying with suggestions, etc. Foster and Dantelian (1966) and Roberts 
(1965) point out tne advantages they see in thb in-basket technique over a case 
study approach, such as providing ,s(5me^ pressure on participants, allowing scrutiny 
in one area while l^eeping a larger system in mind, and placing emphasis on the 
method of solution Vather ttian just the solution. 

That the simulation mode thus initiated met with considerable^ endorse- 
ment is implied by the large number K)f simulations subsequently appearing. Many 
of these were developed by the University Council on Educational Administration 
for the roles of elementary principal, secondary principal, superintendent, 
vocational educator, associate superintendent for business, associate superinten- 
dent for instructi-on and comnunity college president. Other sources have developed 
simulations for training administrators of colleges (Rickard, 1966; White, 1963). 



elementary schoBl principals (Phans, et a}., l%fa), research and developtiiLMit 
managerb {Dillnian and Cook, 1969), counselors (Beaird and Standi bh, h)o'] ; 
and Dunlop and Hintergardt, 1968) and educational evaluators (Worthen and 
Hock, in press). Procedures used have generally followed the in-basket model, 
al^though exceptions can be noted. In addition, numerous "special situation" 
/imulations and siitiulnion games have been devised for planning educational 
systems (Abt, 1967), introducing educational games Hit o the curriculum -- 
Fixit (Gordon, 1968) , evaluating elementary schools -~ the Russel Saye Social 
Relations Test (/Damrin, 1959), providing inservice stimulation for teacners 
-- Pfoject Sesame G (Goodwin, 1966; Goodwin, ejt al- , 1959), conducting pro- 
fessional negotiations (Horvat, 1968), selecting personnel (Mclntyre, undated), 
and training project directors in general knowledge about evaluation 
(Twelker, 1970b, d, e ). 

Simulation materials have also been developed for teachers:; fc^ 
preservice teachers (Lehman, 1966; and Project Insite, undated); for racial 
desegregation and problems of the inner city schools (Cruickshank, et al- , 1967; 
Cruickshank, 1969; and Venditti, undated); and for reading teachers (Utsey, 
et al., 1966). A relatively elaboraft and unique simulation in the area of 
preservice teacher training was undertaken by Kersn and his associates at Oregon 
(Kersh, 1961, 1963; and Twelker, 1957). Via multiple projection modes, using 
films and background materials (e.g., students' cumulative records), trainees 
were placed in the role of a practice teacher in the classroom with a super- 
vising teacner. Filnis of 60 problem episodes were made focusing on the students 
in tne class; the trainee's perspective was supposed to be that of tlie teacner 
reacting to these students. After he reacted to each episode, the trainee was 
presented witn a filmed sequence indicating the probable, or at least one possi- 
ble, outcome of his Decision. Research findings from the project prompted 
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the development of teacher education materials that may be used in a variety of 
less costly instructional settings (Teaching Research, 1968; Twelker, 1970c).; 

As is apparent in the information thus far presented, simulation materials 
are used in many facets of education, one primary area being personnel training. 
However, with the exception of (a) the two sets of simulation materials 
(Worthen and Hock, in press; Hock and Worthen, in press) for training evaluation 
personnel, (b) materials being planned by Teaching Research and tne Special 
Media Institute (Michigan State, Syracuse, Teaching Research and the University 
of Southern California) and (c) a few embryonic development projects recently 
initiated, little in the way of simulation materials has been attempted for 
specifically training personnel in functional roles in research, development, 
diffusion or evaluation. The procedural model that might be followed in design- 
ing such simulations is next considered. 

Proposed Steps in Simulation Des"ic [n^^ 

\ \ Eight sequential steps in designing simulations are outlined and 

\ \ • 

discussed below. . ■ > ' 

V 

I . Qefine the instruct i onal problem, describe the operational educational system , 
Ind interrelate the instructi onal problem with the opera.tional s ystem. ' 

Experts on simulation games often treat this stage of designing an instru.c- 
tional simulation as three independent steps; they are listed as one here to 
emphasize their nearly concurrent occurrence in the planning process In essence, 
this step involves developing the rationale justifying a search for jiew instruc- 
tional procedures; it requires a relatively broad and long-range perspective of 
the substance and needs of the total instructional program. Relatedly, the 



/The discussion in this section draws heavily on the work of Crawford 

and Twelker (1969) and Twelker (1969); the reader is directed to the original 

sources for a more detailed presentation. 
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system in which tne new instructional sequence would be used needb to diS debcriDOd 

thoroughly. Contextual settings in wnich instructional packages are installed 

differ widely on many :>alie. t variables (e,g,, ctiaracteristics of learners in Uie 

target groups, availability of staff and institutional resources)-. When the 

descriptions of the problem and the operational system are related, it may 

become clear that the problem cannot be solved given the operational setting. 

More likely, via several successive approximations, the problem will oe re- 

r 

defined or ctherwise restructured to increase its soluDility/compatibility 
within the operational system. 

1 1 . Specify objectives in behavioral ^terms and develop criterion measur es. 

Again steps listed sequentially by Crawford and Twelker (1969) are com- 
bined oecause of their temporal contiguity. The notion of benavioral or perfor- 
mance objectives is dramatically in vogue at the present time. Althougn not 
wi-tnout limitation, it does seem clearly to represent ar\ advance over the^usual 
state of affairs in instructional sequences. Denoted are both enabling and 
terminal objectives, each of which ideally Should contain information on tfie 
target group, the desired behavior, the givens or conditions of perfoniiance, ana 
the degree of attainment desired. Objectives can be written in the cognitive, 
affective and psychomotor domains (or the recently defined interpersonal and 
regulatTiry domains) (Schalock, 1968). It will De noted below that simulations 
may have a pronounced advantage in "relating to" the affective component of 
^ * behavior. 

Performance measures to indicate degree of attainment of enabling and 
terminal objectives are develcpjed in conjunction witn the objectives, Inter- 
estingly, traditional measurement means can obviously be used to determin- change 
in student behaviors; also, observational and evaluational procedures can oe 
embedded within the simulated situation to add realism to the irjeasuring process 
and hofjefully to enhance the 'validity of the outcomes (essentially an extension 
O of the approach suggested by Schalock, 1969). 

ERIC 
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III.. De termine the appropriateness of simulation . 

This particular stef is of great importance (especially given tlie main 
purpose of this section/ If the difficulty encountered by the authors in 
writing this part is4t all indicative, it also looms as a particularly difficult 
step. It \% at this point that a type of outcome-effectiveness/efficiency 
statement has to be generated for simulation, as well as for available alternative 
instructional strategies. As noted in the first part, simulation has generally 
not been differentially more effective than other instructional modes for imparting 
cognitive knowledge. 

Crawford and Twelker (1969) and Cruickshank and Broadbent (1970) cite, 
advantages and limitations of simulations that bear directly on this issue. 

Simulations seem .advantageous in that they: 

1) Emphasize affective behavior. 

2} Interrelate and integrate affective and cognitive behavior. 

3) Motivate and engage .the learner to initiate and sustain involved 
activity that often seems more relevant than classwork. 

4) Emphasize interaction and serious encounters between the learner and 
a complex, reactive environment. 

5) Empnasize incorporation of the behavior displayed witnin tne personal 

style of the learner. 

6) Permit application of the behavior in a variety of contexts, including 
allowing the novice "to be himself." 

7) Emphasize a realistic "perceptual franie" or relevant set witfiin which 
the learner operates, possibly as ef^^ective in this regard as field traiiiing 
such as student teaching. 

8) Permit more control over what the learner experiences than does tne 
laboratory or field experience. 

/ 
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Conversely, simulations may be inappropriate to the extent that they; 
^ 1) Lack relative efficiency in bringing about acquisition of cognitive 
knowledge. (Note, however, that simulations might be justified in situations 
where learner characteristics render more traditional instructional modes 
Ineffective.) 

2) Cost more in money and devel oprfient time than other instructional modes, 
^ possibly even requiring staff training for their proper use. {Chqjickshank and 

Broadbent, 1970, note, however, that simulations are more economical than scnedul- 
ing ang coordinating diverse laboratory or field experiences for all trainee^)' 

3) Create conditions (e.g., higher noise level, increased student physical 
mobility, etc.) that may "intrude" on more conventional instructional settings, 
particularly with simulation games. ^ 

4) Have as outcomes processes that are not easily evaluated. 

5) Provide feedback (to participant responses) that dften is not em- 
pirically validated or even empirically derived. 

Note that it is p6ssible/for the developer to exit at this point if the 
outcome prognosis fOr simulation is not positive. 

f 

'IV.; Determine tne type of simulation re^cuired. 

If step III above results in a decision to consider simulation furtner, 
the type of simulation that is most appropriate must be determined. Suggested 
as alternative categories by Crawford ana Twelker (1969) are interpersonal 
ascendant simulations, machine/media ascendant simulations, and nonsimulation 
games. (Following our earlier definitions, either type of ascendant simulation 
could be a simulation game if it involved competitive interactions uetween 
players,) The first type, Interpersonal ascendant simulations, are at prui^ent 
more prevalent in education than the other two types; they typically involve 
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player-interaction, role-playing and decision-making. Ttreir prevalence 
probably is due to multiple reasons: their appropriateness for training in 
areas involving human interaction, particularly since the complex branching 
nature of interaction is horrendous to program on a machine; their relatively . 
low cost; their adaptability to varying characteristics of learners; their 
tendency to place much responsibility on the -Reamer-players for the unfolding 
of the simulatedXactivity; and their natural inclusion of timely, relevant 
feedback (usually from other learner-players). 

Media/machine ascendant simulations are those in which a major portion 
of the instructional program is carried by the media or machine. ■ Examples 
wouldjnclude the link trainer, business games that are computer-based, and 
classroom simulations as developed by Kersh (1963). Advantages cited for 
machine/media simulations include: the relative control (even of planned 
variability) and reproducibility of the presentation; teachers are not as 
threatened by the- temporary shift of control to machines (as compared- to the shift 
of control to students -in interperson'al ascendant simulations); the complex 
stimuli can be interjected into the simulation more easily by machine. 

Crawford and Twelker (1969) indicate that a number of business games have 
been developed that are mix-mode simulations (i.e.. media/machine and interpersonal 
elements exist about equally). 

The third category contains games such as Wff'n'Proof. Equations, and 
On Sets which do not attempt to simulate reality. As "intnl lectual games," they 
often have objectives close to standard course objectives and are easily inserted 
in the curriculum. Their development is relatively easy and inexpensive, and 
they are adaptable to single or multiple participants. 
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^ • Develop specificatio ns for the simulation . 

To proceed on this step, the design personnel must have in mind 'both the 



338 



\ 

\ 



learning objectives and a model of reality upon which the simulation will oe ■ 
built (exclpt in the case of a nonsimulation game). The difficulty of achieving 
this step lies in facts that there are no scientifically derived specification 
steps to follow, nor\re there design guidelines, and there are few principles 
for use within particular contexts. This task is formidable but a necessary pre- 
requisite before the following steps can be undertaken: identification of the . 
scope'of the simulation (lo6ation, particular activity, setting, time, causes, 
etk); identification of the\cision-makprs and their motives; and identification 
of interactions between decisionmakers and other operations, including feedback 
and information flow. 

VI. Develop a prototype of the simulation system, . 

Logically following v'. this step should proceed without major difficulty 
if the prior step were accomplished satisfactorily. Minor alterations in spec- 
ifications and decisions will have to take place during this step, but effort . 
required will be minimized to the extent that efforts on Step V were full and 
complete. 

VII. Pilot the simulation system -prot ot ype and modify according ly. 

Regardless of the skill with which the prior steps have been carried out. 
Step VII is essential. Designers of the simulation must observe in* detail a 
small-scale tryout with a limited number of students. Crawford and Twelker ,(1969) 
wisely suggest video-taping the tryout so that ■4difi cation plans derived can be 
checked against transactions that occurred; an accurate recording and observational 
system would be a logical alternative if video-tape was not available. Note, too. 
that changing the simulation to make it more effective is only one alternative 
open to the designers; a decision could also be made to discard the simulation 
if it does not, to some marked cfegree, bring about the pre-set objec^tives. 
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Vin. Conduct the field trial and make further modifications . 

The modified simulation should have a full-scale field trial- Supports 
available previously (e*g*, designer present during tryout and easy modification 
of procedures) are no longer present; the simulation must stand on its own* 
Again* an accurate recording system is in order to gather data that can be used 
in making essential modifications. Outside evaluators are sometimes utilized, 
as the designer is committed'by tlTis point to a degree where hi,s complete 
impartiality would be difficult if not impossible^ Obviously, final refinements 
before implementation and dissemination are made on the basis of data collected 
during the field trial; including opinions of the stud^ts of the simulation. 
Some designers incorporate into their "final" product, procedures by which 
infonriative data come back to them from users so that future modifications or 
changes can be data-based. 

8 

In a recent communication, jt has been indicated that modifications have 
been made in the material from which the above steps were adopted and will soon 
be published. *Early in the sequence, an additional necessary step appears to 
be defining the management and staff resources available to undertake the 
simulation development effort. Also, the "criterion measures" portion of Stfep 
II has been separated out and included in a later step that involves valid- 
ation of the simulation System. However* despite further modifications, the 
steps presented her^ should be useful in guiding beginning efforts. 

With this general model depicted, let us now turn to conside,r simulation 
with specific regard to resea'^ch, development, diffusion and evaluation training. 



8 

Twelke?^ P. A., Personal communication, July, 1970. 
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Amenability to Simulation of Areas in Research , 
Development, Diffusion and Evaluation Training 

In developing this paper, it was initially planned to merely list areas 
wit.hin research and research-related role training that seemed to lend them- 
selves "naturally" to simulation as an instructional technique. This intended 
procedure was abandoned for the most part because the approach Tacked supporting 

rationale. * 

The format subsequently decided "upon and used nere is much more 
systematized, but nevertheless is highly presumptive. and open to question^ 
Keying on the earlier' work of tHe Task Force (see Chapter 2), skills listed under 
the functions of research.," research-based development, diffusion and evaluation 
were considered individually in relation to simulation. Specifically, eacn skill 
was judged as to: (a) Us susceptibility (great, moderate or little) to simulation; 
(b) the probable 'amount of work (great, moderate or little) that would be^ required 
to develop simulation materials for use in teaching tnat skill; and (c) the 
present availability (available,' in progress or unavailable) of simulation 
materials bearing on the skill. The tables following were subsequently derived. 

The judgments reflected in the tables that follow were made by tne autnors, 
with the assistance of a small number of cons-ultants . Although not entirely 
Arbitrary, many of the decisions about classification were oased on available 
" information and on the slimmest of "evidence." Specific reservations aoout the 

tables are listed below: 

1) There is little empirical support available for decisions in the table. 

2) The skills on which these tables are based are listed in a type of 
chronological sequence, e.g.. as they would be needed in the course" of a research 
study. Much more appropriate would be a hierarchical analysis organizing those 
skills and making explicit the subskills involved. 
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3) Certain of the skills are listed in such a manner that it is not 
apparent whether they are at a knowledge level or an application level. 
Judgments about their susceptibility to simulation are thereby more difficult., 

4) Certain skills might be judged to, have pronounced susceptibility to 
simulation, yet developing a simulation relating to the skill mignt not bo 
rational, since there are much more simple and direct ways to teacn tne skill, 

5) Listing susceptibility to simulation for each individual skill mignt 
be misleading and overly fractionating; that is, a single siftn^lation could be 
designed to reUte to several of the skills. 

Therefore, the reader is cautioned appropriately to view the taoles as 
a first generation effort that requires considerable refinement and empirical 
.validation. One approach would be to ask for similar judgnents from a much 
larger group of research and research-related personnel; Ariotner validation 
technique might be to survey a small sample of research and research-related 
personnel, having them denote what were the severest problems faced oy them in 
the field, and the skills required to solve these problems. An instrument 
listing these problems might then be sent to a larger sample to determine fre- 
quency of occurrence. From the resulting list o/ severe problems frequently 
occurring (getting at a priori ty' dimension not currently represented in tne 
tables), a panel of judges might be asked to select the alleviating skills most 
anjenable to simulation, - . 

Regardless of subsequent steps taken, the tables^ presented here should 

be considered highly tentative. However, they^should be useful for making molar 

9 

decisions about areas in which simulation development mignt oe undertaken. 



9 

It can be noted that this procedure essentially '^jumps" to Step III in 
simulation design as presented In the second section above* This was done 
because of the great number of contextual situations that might and do exist and 
that would be comndered in Steps II and III. In unusual contexts, certain of the 
judgments presented in the tables might be altered. 
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Table 5.2 

Potential Utility of Simulation for Training in Research. Ski 11 s 



Skill 



Susceptibil ity 
to 

Simulation ' 



M - G 



G 
G 

M - G 
M - G 



1. Drbwing implications from prior 
research 

2. .Identifying and delineating signifi- 

cant researchable problems,. 

3. Procuring research resources 

4. Managing research resources 

5. Interpreting, evaluating and synthe- 
sizing relevant literature 

6. Formulating hypotheses- to be tested 
in the study 

7. Specifying data necessary for rigorous M 
test of hypothesis 

8. Identifying population to which results 
should be general ized 

9. Using appropriate sampling techniques M - 6 
to draw a sample , 

Understanding experimental and other L 
systematic approaches to inquiry 

Drawing on knowledge of 10 to design 
a study appropriate Tor the problem 

Applying the rasearch design recog- 
nizing and acting on threats to 
val idity 



10. 
11. 
12. 

13. 

14. 

15. 

16. 
17. 

18. 



M - G 



M - G 



Identifying behavioral outcomes for 
measurement 

Selecting specific variables and . 
treatments to be used 
Selecting appropriate techniques of 
measurement ^ 

Developing measuring instruments 

Asse$sing the validity of outcome 
measures 

Using a variety of data gathering 
methods 



L 
M 



Probable 
Amount of 

Work 
Required 



M 

6 

G 
G 
M 

M 
L 

m' 

G 

M ' 
M 

M 
G 
M 

G 

G 

M 
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^Contijiued}^ 



Present 
Availabil'i ty 
of Simulation 
Materials 



IP" 
U 

A 

IP' 



b. 



U 
Li 
U 

u 



A^' 
U 



Ij 

u 
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Skill 



Susceptibility 
to ' 
Simulation 



Probable 
Amount of 

Work 
Required 



Present 
Availability 
of Simulation 
Materials 



|.j?i;?irig data for analysis L 

• if.derstanding types and assump- L 
underlying various statistical 

{ * thnu|ues 

brjwmg on knowledge of 20 in select- M - G 
incj appropriate data analysis tech- 

aids in data processing, L 
^ucr. as a cwputer 

i i LHrpreting and drawing conclu- M - 6 

-ijns from data analyses 

i nnuiating theoretical statements L 
atjitt the studied behavior 

^fp '-ring research findings, orally L 
■ •'^•i if* vvritihg 



L 



M 

M 
G 
M 



U 
U 

u 
u 



. f\owin & J. Miilman, Cornell University 

r_:l Market Place, R. Horn, Information Resou :es Inc,, Cambridge, Mass, 
rir.'j R&D Project Managers, D. H. Dillman & D. L. Cook, Ohio State Univ. 
: J.. Crawford, Jefferson, Oregon 



L 1 1 L 5 e 



A - Available 
IP - In Progress 
U - Unavai lable 
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Tdble 5.3 

Potential Utilicy of Siiiiulation for TrainiiKj in 
Research-based Development Skills 



Skill 



Susceptibil ity 
to 

jimuldtion 



Probable 
Aiiiount of 

Work 
Roqui red 



Present 
Availabil i ty 
of Simulation 
Material s 



Iv Interpreting information concern- ' 
ing educational goals 

2., Orawing on research results in plan- 
ning developmental activities 

3. Conceptualizing systems, their 
elements, and interrelationships 
among elements 

4. Specifying desired performance out- 
comes of instruction 

5. Devising techniques to identify* 
entry capabilities of learners 

6. Identifying alternative instruc- 
tional and media techniques 

7. : Determining appropriate sequences 

of topics in instruction 

8. Describing the product to be 
developed 

9. Constructing effective oral and 
written forms of instructional 
communicati ons 

10. Directing the work of prcducllon 
' personnel 

11. Selecting or devising appropriate 
techniques for measuring outcomes 

12. Designing initial laboratory tests 

of developed techniques and materials 

13. Managing initial laboratory tests of 
developed techniques and materials 

14. Designing field tryouts and test: 



G 
G 
G 
G 
G 
G 

fi 
M 

M 
G 
M 



M 
M 
G 

M 
M 

G 



M 
M 



IP" 
U 

ipb 



u 
u 



(ContinuedJ 
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Potential Utility of Simulation" for Traimiiq in 
Research-based Development Skills 

(Cont.nued) 



Skin 

15. ; Managing field t^youts and tests 

16. Reporting evaluation of outcomes 

17. Interpreting evaluation findings 

18. ^ Specifying revision requirements 

based on outcome evaluations 



Susceptibi 1 ity 
to 

Simulation 



G 
L 
G 
G 



Probable 
Amount ot 

Work 
!U.H]Uir(^d 



M 

M - G 
M 



PrescMit 
Availabil 1 ty 
ot Simulation 
Matv^ri.ils 



^P. A. Twelker, Special Media Institute Teaching Research, Monmouth, Oregon 
^Special Media Institute, Teaching Research, Monmouth^, Oregon 

G - Great . ( A - Available 

M - Moderate ^ I ' IP - In Progress 

L - Little \ U " Unavailable 
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Potential Utility of Simulation for Training in Diffusion Skills 



Skill 



Susceptibi 1 ity 
to 

Simulation 



Probable 
Amount of 

Uork 
Rcqi! i red 



Present 
Availability 
of Simulation 
Ma terials 



Diss enii nati on 

1. Defining and an.-'.lyzing charac- 
teristics of target groups 

2. Selecting from developed packages 
those most effectively disoennnable 

3. Selecting most effective dissemination 
vehicles for target groups 

4. Composing information for accurate 
and pervasive dissemination 

5. Implementing actual ciissemination 

6. Directing techn-cal production 
personnel 

7. Designing techniques for evaluating 
dissemination effectiveness 

8. Implementing design for evaluating 
dissemination effectiveness 



De monst ra tion 
1 



Specifying natui'e of demonstration 

2.. Selecting appropriate setting for 
demonstration 

^ 3. Selecting appropriate personnel 
for demonstration 

4. Managing and coordinating the 
demonstraiion offort 

5, Evaluating the daiionstra tion ' s 
effectiveness 



G 
G 



G 
0 



M 

M 

M 

M 

M 
M 

M 

M 

L - M 
L - I-l 

L 

M 
M 



II 
II 
U 
U 

u 



u 
u 

,u 

u 

u 



( Cont i nue dj_ 
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Table 5.4 

Potential Utility of Simulation for Training in Diffusion Skills 

(Continued) 
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Skill 



Facil itating Adoption 

1. Analyzing differences between 
adopting organi^a-tsoajnd 
development^^ 

2* Designing/^oduct modifications 
to fit adopting organization 

3. Designing adopting organization 
modifications to fit product 

4. Training personnel in adopting 
organization 

5. Identifying potential barriers 
to implementation 

6* Devising long-range evaluation of 
the installed package 

1, , Conducting long-range evaluation 
of the installed package 



Susceptibility 
to 

Simulation 



G 
G 
L 

G 
M 
G 



Probable 
Amount of 

Work 
Required 



L - M 

M 
M 

M - G 
M 
M 
M 



Present 
Availabil ity 
of Simulation 
Ma terials 



G - Great 
M - Moderate 
L - Little 



A - Available 
IP - In Progress 
U - Unavailable 
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Table 5.5 

Potential Utility of Simulation for Training in 
Context Evaluation/Situations Analysis Skills 



1. 
2. 

3. 

4. 



6. 



8. 



10. 



11. 



12. 



Skill 



Susceptibil ity 
to 

Simulation 



identifying goals of the system 

Assessing the social relevance 
of these goals 

Identifying values that are implicit 
in the systan goals 

Identifying standards decision- 
makers use in interpreting data provided 

Clarifying and explicating desired 
outcomes of the system 

Measuring current actual system out- 
comes through demographic analysis 

Measuring current actual system out- 
comes through economic analysis 

Measuring current actual system out- 
comes through psychometric analysis 

Measuring current actual system out- 
comes through systems analysis 

Measuring current actual system out- 
comes through observational techniques 

Comparing actual with intended system 
outcomes to identify discrepancies 
and needs 



M - 6 
M 

L - M 



M 



M - G 



M - 



Explicating problems creating the 
needs and diagnosing causes of ■ "' 
the problems 

13. Assisting system personnel to develop 
objectives to satisfy needs or solve 
problems 

14. Designing a monitoring system to 
provide feedback on the operating 
System 



M - G 



probable 
Amount of 

Work 
Required 



G 
G 
M 



G 
G 
G 
G 
G 

L - M 



L - M 



Present 
Availabi 1 i tv 
of Simulation 
Materials 



i^a.b 

\^ 

u 
u 

^a,b 
U 

u 

u 
u 

/^a,b 
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^ B. R. Worthen & M. D, Hock, Worthington, Ohio: Charles A, Jones Publishing 
Co. , In Press 

^ M. D. Hock a. B. R. Worthen, Worthington, Ohio: Charles A. uones Publishing 
Co. , In Press 

G - Great A - Available 

M - Moderate IP - In Progress 

L - Little U ~ Unavailable 



Table 5.5 
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Potential Utility of Simulation for Training In 
Program Planning/Input Analysis Skills 



Skill 



Susceptibil ity 
to 

Simulation 



1. Assisting system personnel L - M 
determine operational feasibility 

of proposed objectives 

2. Assisting system personnel establish L - M 
priorities for the selected objectives 

3. Identifying and rating alternative L - M 
strategies for attaining the 

selected objectives 

4. ^ Identifying auid rating available L 
' resources for support 

5. Selecting an implementation L 
^ strategy 

6. Selecting support source! and M 
resources to be used in implementing 
program 

7. Predicting potential barriers to M r G 
success and strategy's potential 

for overcoming them 

8. Identifying mo^t-likely-success- ,M - G 
ful tactits to implement selected 

strategy 



Probable 
Amount of 

Work 
Required 

L - M 



L - M 
L - M 

L 
L 

M 



Present 
Availability 
of Simulation 
Materials 



xa,b 



^a,D 



^ B. R. Worthen & M. D. Hock, Worthington, Ohio; Charles A. Jones Publishing 
Co., In Press 

^ ^M. D. Hock & B. R. Worthen, Wopthington, Ohio: Charles A. Jones Publishing 
Co., In Press 

^ - Great 
ft - Moderate 
L - Little 



A - Available 
IP - Inr Progress 
U - Unavailable 
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Table 5.7 

Potential Utility of Simulation for Training in 
Process Evaluation/Progrcim Monitoring Skills 



Skill 



Susceptibility 
to 

S imulation 



1 . 
2, 

3, 
4, 

5. 



De^iigning and selecting indicators 
of progress in educational progress 

Monitoring the program using 
multiple techniques to detect 
deviations from design 

Anticipating barriers and remain- 
ing alert to unanticipated problems 

Providing immediate feedback to 
decision-makers for their use 

Perceiving human relations problems 
that threaten the program 



M 



Probable 
Amount of 

Work 
Requi red 



M 
G 

M 
M 

M - G 



Present 
AvailabilUy 
of Simulation 
Materials 



A 



II 

a ,b 

^a,b 
.a,b 



^B, R. Worthen & M. 0. Hock, Worthington, Ohio:; Charles A. Jones Publishing 
Cc. , In Press 

^M. D. Hock, & B. R. ^^rthen, Worthington, Ohio: Charles A, Jones Publishing 
Co. , In Press 



G - Great 
M - Moderate 
L - Little 



A - Available 
IP - In Progress 
U - Unavailable 
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Potential Utility of Simulation for Training in 
Outcome' Evaluation Skills 
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Skill 



Susceptibility 
to 

Simulation 



1. Applying appropriate designs 
to evaluation studies 

2. Designing criteria and data 
collection procedures to measure 
the effectiveness and eTficiency 
of existing innovative practices 
and products 

3. Translating objectives into 
behavioral terms, if necessary 

4. Identifying situations where 
designated behavior can be 
observed and recorded 

5. Establishing standards for judging 
whether objectives have been attained 

6. : Selecting or developing measure- 

ment techniGues to yield data 
bearing on standards 

7. Assessing the validUy of outcome 
measures 

8. Collectiny data prior to analysis 

9. Organizing data prior to analysis 

10. Selecting an appropriate technique 
to analyze data 

11. ; Analyzing the evidence yielded by 

evaluation 

12. Judging strength of plans and 
procedures employed to meet 
objectives 



L 

M 

M 
L 



L 
L 

G 

L - M 
M 



Probable 
Amount of 

Work 
Required 



M 
M 



L 
M 

M 
M 

M 

G 
M 

M 

M 
M 



Prevent 
Aval lab 1 1 i ty 
of Simulation 
Materials 



A 



a ,c,d,e 



,c,d 



,c,d 



.c,d 



A' 



a , c , d 



A" 
U 

a' 
u 

I) 



(Continuedj 
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Tab^e b.8 



Potential Utility of Simulation for Trainin^j m 
Outcome Evaluation Skills 



Skill 



Susceptibility 
to 

Simulation 



13. Explaining the outcomes as a function 
of plans, procedures, and resources 

14. Deciding upon recommendations 
based on outcomes 

15. Estimating impact of outcomes 
on problem area being served 

16. Providing information to allow 
-decision-makers to continue, 

modify, or terminate program 

17. Specifying needed changes in context 
evaluation system due to decisions 
about program continuation 



M 
L 
M 



Probable 
Amount of 

Work 
Required 



G 

M 
M 



Present 
Availabil ity 
of Simulation 
flaterials 



A 



c,d 



^Impact Evaluation Game, P. A. iwelker, Teaching Research, Monmouth, Oregon 

'^R. Harmiond, EPIC, Tucson, Arizona; Special Media Institute, Teaching 
Research, Monmouth, Oregon 

^B. R. Worthen & M. n. Hock, Wor'thington, Ohio: Charles A. Jones Publishing 
Co., In Press 

^M. D. Hock & B. R. Worthen, Worthington, Ohio: Charles A., Jones Publishing 
Co., In Press 

^UCLA Center for S^tudy of Evaluation and Instructional Programs 



G - Great 
M - Moderate 
L - Little 



A - Available 
IP - In Progress 
U - Unavailable 
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Important Issues Involving Siinulation 1n the Trdlmruj 
of Pers ons in Researcn and Research-related Fu nctions 

The basic issue, mentioned previously, is the extent to which simulation 
i's an appropriate instructional mode for training in research, development, 
diffusion and evaluation. While a direct answer to this query is not possible, 
three positive factors can be pointed out. First, several of tne advantages 
listed in the second section of this paper for simulation as an instructional 
technique are nearly unique to simulation. For example, simulation would allow 
the trainee to be exposed to a variety of realistic situations, involving him 
by requiring decisions on his part. Comparable realism would surely De as 
obtainable (even more obtainable) in an actual field situation, but exposure 
to a variety of such experiences in a relatively short time period under 
"controlled" conditions would not. A second factor bearing on this issue 
is that a limited number of simulations has been developed in this area, and 

developers and users almost unanimously report satisfaction with the perfomrance 

10 , ^ f 

of these products. Finally, there is an apparent need for the development t>r 

viable alternative training procedures in this area; simulation seems to qualify 

as an alternative of some potential. Thus, although not directl^y resolving this 

issue, the authors feel warranted .in concluding that there should be supporl: for 

a sustained developmental effort in! research and research-related role training 

to determine the operational strengths and weaknesses of simulation. 

Considered below are other issues grouped into development issi/es and 

administrative issues. No attempt has been made to be exhaustive; several 



^"^Of course, such satisfaction may be true of developers of most- 
products, but satisfaction of users is a. more rare commodity. 
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development issues raised by Cruickshank and Broadbent (1969) are not included' 
(e.g., game quality, feedback and realism), nor are Issues related to simulation 
instructors (e.g., role of the director) or simulation evaluators (e.g.= . specificity 
-of outcomes' and transfer of training). ■ The 'devglopinent issues that are considered 
here are not considered elsewhere for the most part; they reflyfect the authors' 
thinking on some of the apparently' more critical issues. 

A key issue in development is the' lack of basic research that bears on 
the development of simulation. This type of research has'been neitner funded 
nor undertaken independently. There is little agreement as to the desicjn para- 
meters that such research should have; there is little consensus as to which 
are the crucial variables to study, or as to which dependent variables to use. 
Thus, this lack of a programmatic research effort on matters related to simulation 
creates a number of problems and uncertainties tnat tne developer must face. 

A second issue in development, related to the first, revolves about tne 
relative effectiveness and efficiency of simulation compared to other instruc-, 
tional alternatives. Hard data on this issue essentially do not exist. For 
^example, in the area of simulation games, few games lend themselves well to 
researching their own effectiveness/efficiency (and essentially none were devel- 
oped primarily for research); often the pronounced differences permitted in game 
administration are more than sufficient to mask effects of studied variables^on 
game outcomes. Important questions about the -relative effectiveness and efficiency 
of simulation experiences compared to internships/apprenticeships have not Deen 
answered, e.g., there "is no empirically supported estimate of the extent to-which 
an effect during a simulation approximates that experience in the related phase 
of an internship. 

Another issue concerns how the simulation developmerit is to proceed. 
Guidelines for simulation design and development that have been extensively and 
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empirically validated are unavailable. There are very few "if-then" 
principles tnat have been derived in this area; little has been^done to link 
design format to the context in which the simulation will be used. Particularly 
missing are agreed upon standards' for the evaluation of simulations; procedures 
have not been established to decermine the external validity of simulations 
(in the absence of the developer, certain simulations have been known to operate, 
poorly); and little information exists bearing on the degree of acceptability 
of simulation in differing operational settings,' 

The issue of resource cost of tne simulations looms as a particularly 
critical one/ This includes both monetary and time costs during development 
periods and subsequently when the simulation is being used, Tnese co^ts are 
essentially unknown and should not be underestimated; properly developing a 
simulation can be a costly and complex undertaking, and in operational settings, 
staff beginning to use the simulation who are unfamiliar with the technique may 
need inservice training. The coirparative costs of different types of simulation 
(interpersonal ascendant, machine/media ascendant, and nonsimulation games), 
during both development and use, must also be considered. 

Other development issues center around the possible uses that can be 

made of simulation. Already alluded to has been the matter of wtiether simulations 

J- 

can be used as appropriately for assessment as for instruction. How appropriate 
is it to think of developing simulation^ to train the educator to be a wise 
evaluation-consumer or^ research -consumer? v 
Anotner development issue is /whether the simulation snould be designed 
to replicate a portior\ of, or a total, system (e.g., in researpher training, 
Whether the simulation involves only applying for grant funds or vfnether it 
involves carrying out several elements of the study, from securing resource*s 
to data interpretation); It can be noted that a series of unrelated, snort 
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simulations might take on the characteristics of exercises ;aod, as slicn, be less 
likely to generate the involvement or affective potential that a longer, articu- 
• Uted simulation would'. In this connection, 1t can be speculated (as noted 
above, m hard data are available on this issue) ^that very few field internships 
provide experiences characteristic of those required of the "whole" researcher. 
Allowing for exceptions and depending upon the site and'^ength of the internship, 
the trainee is likely to experience only the constj^uction of measuring instruments, 
or only oKta collection, or only data scoring, or only .data analysis, etc. 
Realistically, trainees normally seem to get a massive dose of training in one 
or a few skills and little else. Opportunities to worlc with a "master" or even 
an experienced researcher are limited, at best. A carefully worked out simulation 
could provide a series of integrated experiences in a short time period, 
experiences that would-not be duplicated-in most internship situations. (Note, 
though, that such simulations ordinarily would be u^ed in addition to intern- 
ships rather than in lieu thereof; internships provide learning experiences 

of other kinds.) • 

Turning from development of simulations to a brhf consideration of . 

administrative issues, one major issue involves the incorporation of /the 

I 

simulationCs) into 'research training programs. Whether they are better used 
for orientation or for synthesis near the end of tracing (see La Grine, 1964)- ^ 
is an open question. Gagne' (1.962) feels that simulatj/6ns are more appropriate 
for" later than for earlier learning. On the other band, simulations might 
\ make beginning trainees feel mOre involved, more quickly, in educational 
research (possibly helping to reduce the number of capable trainees who 

• drop out, etc. ) . • ^ 

Other administrative- issues include how long the simulation should bp, • 
whether practice is spaced or massed, and the size of the group involved in 
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the simulation (Cruickshank and Bro^dbent. 1970). Expanding somewhat on the 
last issue, it can be observed tnat by designing and conducting certain 
simulations for teams, and by placing on teams (with the trainees) educational 
practitioners of various types (e.g., superintendents, principals, supervisors, 
teachers and board members), the trainees might gain some tri-ghly maturational 
"hard knocks'' in the security of a simulated school situation. Such experiences 
might make them more knowledgeable in important humanistic ways (e.g., learning 
of constraints on persons filling these educational roles, iheir motivations, 
their perceptions of university researchers)* 

Related to the issues of spacing of simulation experiences and length of 
the simulation, and noteworthy, is consideration of the merit of a simulation , 
carefully designed and validated to be used in a controlled, sequestered 
:>etting; i.e.,^it would be a "live-in" simulation conducted at a simulation 
training center. The control possible in an intensive live-in experience, 
say, of a week's duration has much.to be said for it, particularly when many 
well developed products fail when used by persons lacking specific training in 
their use. There would be a continuing role at such a center for trained 
instructors, and a role for R, D, D and £ exi)erts (both in tne design and 
validation of the simulation and as a jury during early Lryouts of tne simulation 
at the center). And how often is U possible for trainees to oe exposed to tne 
thinking of a master researcher? Given appropriate staffing, such a center 
might make this possible. The intensity and qualUy control possible at sucn 
a center would suggest that due deliberation and consideration be given to such 
an undertaking.: 

Suffice it to say, from the issues considered briefly above, that e 
great number of issues relate directly to simulation. Tfiese issues and otners 
should be made even more explicit for those rontemplating the deVeloprnent or use * 
of simulation in tne training of evaluators, diffuse>^s, developers or researcners. 
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Sufffnary 

This section has included (a) a general review of the use of siiiiulation 
as an instructional aid, (b) suggested steps in developing simulations, (c) a 
listing of research and research-related skills that might be taught through 
simulation, and (d) a discussion of some issues that are in need of resolution 
before simulations can be confidently developed and adini ni stered in programs 
designed to train research and research-related personnel. Much work needs to 
be done before precise specifications can be outlined for use in developing 
relevant simulations for training such personnel. In the meanwhile, it is 
the opinion of the authors that the technique holds sufficient promise to 
warrant attempts to develop, use and evaluate it as an alternative method of 
training educational research, development, diffusion and evaluation personnel.- 



ERIC 



CHAPTER 6 

SUHMARY, CONCLUSIONS AND RECOMMENDATIONS 
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SUMMARY, CONCLUSIONS AND RECOMMENDATIONS 

The activities reported in the previous chapters were undertaken 
in response to an obvious need for a coordinated approach to the training 
of research and research-related personnel in education. .The expertise ^ 
possessed by such personnel is essential in building a knowledge base 
which will make possible the massive improvement needed in American 
education. In this initial project, therefore, the Task Force concentrated 
on furnishing pre*liminary information xofteefnTng the kinds of knowledge 
and skill require4H3y"fesearch and related personnel and baseline data 
on programs for trainirig in educational research and research-related 
areas. It should be stressed that this is a report of research jnd 
developmental efforts ^er se_and, consequently, the conclusions and 
recotmendations included in this chapter are restricted to those that 
can be inferred rather directly from the data and procedures discussed 
in this report. Task Force deliberations have resulted in other 
positions and recommendations that are based more on collective judgements 
than on ddta; these position statements will be presented later in a 
separate ■ocument ( Technical Paper No. 18, to be distributed in February 
of 1971). 

Summary and Conclusions 

Four major areas of investigation were undertaken in thp course 
of this study. The first was the development and testing of a classifi- 
cation system of skills necessary for the conduct of research, development, 
'diffusion and evaluation in education. The second was an^an'alysis of 



the AERA employment service data from 1968, 1969 and 1970, with special 
attention to the demand for and availability of research and research- 
related competencies. The third study was an analysis of characteristics 
of trainees in Graduate Research Training Programs funded under Title IV 
of the Elementary and Secondary Education Act of 1965. The fourth 
effort involved examination of alternative approaches to the training of 
educational researchers in specific areas'of competence. 

Classification System for Research-related Skills 

The development and testing ST, a classi figation system for 
research and research-related skills in -education was reported in 
Chapter 2. Initial discussions among members of the Task Force resulted 
in the listing^f several functions related to educational inquiry; these 
discussions were followed by individual conceptual efforts which yielded 
lists of skills required for the several functions. The skill lists 
were modified and organized into a classification system which was then 
tested in telephone interviews on a sample of 58 employers or supervisors 
of educational research and research-related personnel in ten institutional 
settings. 

Results of the telephone Interviews showed that the most important 
skills and those which are in shortest supply are skills involving, in a 
broad sense, identification and description of problems and goals, 'and 
eu'iiuation of success in solving those problems and reaching those goals. 
It is accurate to say that the skills which are most urgently needed 
are those requiring the exercise of judgement, and not simply the applica- 
tion of standard methods and techniques to standard problems and situations 
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Ifwould appear, in other words, that the most needed skills are those 
which, in the opinion of the interviewees, are hardest to train for. 
This conclusion is evident in Appendix D, where the interviewees' 
opinions regarding the level of difficulty involved in training for the 
needed skills are reported. Nearly all of the skills requiring judge- 
ments were categorized as requiring long term train1i:y, and in many 
cases an internship/apprenticeship/practicum experience was suggested 
by the interviewees. There were also frequent references to the fact 
that on-the-job experience and exposure to real problems are necessary 
for the development of some of the most important skills. 

In the final section of Chapter 2, the classification system 
was re-examined and the skills required for educationa,l research and 
educational evaluation were reconsidered. Under the research heading, 
thirteen major skills were identified and their importance was discussed 
for three separate types of educational research: empirical, behavioral 
research; empirical, nonbehavioral research; and research on methodology.. 
For educational evaluation, ten general skill areas were identified as 
important. A discussion followed of essential knowledge of methodology 
required by educational researchers. It was held by the authors of this 
section that many educational researchers may not need as detailed 
knowledge about each specific research technique as is often suppobcd; 
on the contrary, they need to have only a working knowledge about each 
area of research methodology relevant to the discipline within which 
they work. In-depth knowledge of the discipline is a more critical 
concern and, in and of itself, demands extensive preparation. It was 
argued in this section that only vor specialists in particular techniques 
{e.g., statisticians) would one need to go beyond mastery of these essen- 
tial skills and content of the relevant discipline. 



It was noted at the conclusion of this section that, until 
!nore effective ways to train persons and inculcate in them the necessary 
skills are developed, such skills might best be learned through apprentice- 
ship training of some type* At present, direct teaching of many necessary 
skills is not included in formal coursework and it appears likely that 
it will be some time before techniques are developed to effectively 
transmit many of these skills in the classroom. In the interim, it may 
be not only desirable but necessary to depend on apprenticeship training 
to transmit many important skills. 

AnaJy^s ^o^EI^E^mployment Serv ice Data 

In an effort to provide infomation on the relative demand for 
and availability of research skills and areas of competency, the Ta<:k 
Force examined the AERA employment service records for 1968, 1969 and 
1970 (see Chapter 3). For each year, the competencies required by 
employers for specific job openings and those listed by applicants for 
positions were compared and apoarent discrepancies were noted and 
discussed*. 

In addition, for the 1970 employment service the Task Force 
adr>inistered a checklist containing 39 skills derived from the original 
classification system. Employers and applicants were asked to indicate 
the degree of skill required or possessed for each item, and chi -square 
tests were run to determine the significant differences between employer 
and applicant responses, and between university and non-university 
employer responses. Although several items did yi^^ld significant values, 
no pattern was evident in the differences between employer and applicant 
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respunses. In the other between-groups comparison, it was found ttiat 
evaluation^kills are required in a higher degree for non-um versi ty 
positions than for those inside the uni versi ty. 

Overall the errtployment service data for the three years indicate 
the following;^ 

1. Whereas the number of applicants remained relatively stably 
from 1968 to 1970, the number of posit' ons has declined markedly; the 
ratio of applicants to positions rose from 1,17:1 in 1968 to 1,76:1 in 
1970, It was noted, nowever, that if those positions (both those 
available and thpse sought) which are classified as nonresearch are 
eliminated from consideration, then the ratio of applicants to positions 
is about the same for 1968 and 19V0— roughly two to one" each year. 

2. In 1970 (as gjmpared 1;q 19(58) applicants had dramatically 
fewer potential positions to choose from in competence areas such as 
educational research, research design, survey or institutional research, 
evaluation of instructional products, instrument development and 
construction, and elementary and advanced statistical techniques. 

3. Results from the 1970 telephone interviews (reported in 
Chapter 2) revealed many competence areas that are considered both 
important and in short supply by project directors, research organizatioi 
heads and other employers of research and research-related personnel. 
There was only limited correspondence, however, between those competence 



^It should be noted that it is not known whether these findings 
are true for the educational research conmunity in general or only for 
that portion of it represented by users of the AERA employment service. 



areas and the skills which were in most demand at the 1970 AERA employ- 
rnent service. Two possible explanations were offered. Pernaps the 
relatively recent budget restrictions placed on program directors and 
other employing organizations are such that their critical personnel 
needs are not accurately reflected in vacancies on the job market. 
Or perhaps the lack of correspondence represents an implicit criticism 
by employers of the level of competence of present pr available personnel 
in certain important sl^ill areas. 

4* In none of the three years was there evidence of the large 
numbers of vacancies in the areas of educational development and diffusion 
that had been predicted by Clark and Hopkins (1969). 

5, Examination of the data concerning geographic location showed 
that there were marked discrepancies between the number of posi^^ions 
available and the iiumber of applicants preferring positions in^the 
South, the Midwest and the Pacific/West Coast regions (fewer applicants 
than openings in the South and Midwest, and more applicants than openings 
in the Pacific/West Coast region). Overall, however, a very large 
proportion of the applicants expressed no strong geographic preference 
at all . 

Ti tle IV. Graduate Research Training Programs 

As a first step in detern^ining what research skills and owledge 
are being learned in formal training programs, the Task Force undertook 
to describe the graduate research training programs funded under Title IV 
and to compare the 1969-70 trainees with those studied earlier by Sieber 
(1968) for 1966-67. The following observations were noted in Chapter 4. 
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1- While the characteristics of the 1969-70' trainees were 
similar to those of the 1966-67 group, tnere were some Important 
differences: The 1969-7C group was somewhat younger and more of them 
were recruited directly from their previous degree program; this 
group also had a broader disciplinary base and tended more often to 
be seeking the Ph.D. rather than the Ed.D. 

2. The training programs, for the most part, were located at 
good research institutions and included a strong interdisciplinary 
emphasis; the trainees were very talented academically, receiving GRE 
and MAT scores on a par with or higTler than those of a majority of 
students in virtually every profes<;ional and substantive field referenced, 

3. Although the trainees, the program directors and the research 
units in which the programs are located have been well described, the 
actual experiences of the trainees in the course of their work -- i.e,, 
the substance of the Title IV programs ~- are not adequately known; thus 
it is difficult to know whether the graduate research training programs 
are providing the skills and knowledge which are essential for the 
conduct of educational research. 

4. On balance, the Title IV programs are deserving of strong 
continuing support. 

Alternative Approaches to Educational__Researxh_l>aini^ 

Chapter- ^ contain: a discussion of alternative approaches to the 
training of educational researchers. The discussion is in three parts;:: 
(1) a survey of inservice training programs of professional organizations 
in areas outside educational research, (2) a brief description of 
training pocsibil i ties stimulated by the Task Force, and (3) a detailed 



examination of one of the training techniques proposed as a result of 
Task Force solicitation* 

Examination of the inservice training programs of six professional 
organizations showed that the most common training vehicle is a traveling 
course of one to three days' duration. Jhe course subjects are 
generally solicited informally from the membership or selected from 
current literature, and the course content is developed within the 
organization. Participants are selected through self-screening and a 

« V 

tuition fee is usually charged. It was concluded that some elements 
of the training programs examined held promi^for future AERA educa- 
tional research training efforts. 

Investigations of three training possibilities were stimulated ^ 
by Task Force activities. The first of these is the development of a 
library of cassette tapes on current research topics. The second 
involves transportable packages of training materials which might be 
developedVroni existing AERA presession materials. The third is a 
consideration of the uses of simulation techniques in the training of 
educational researchers. 

The possibility of applying simulation to educational research 
training was examined in detail in Chaper 5. Consideration was 
given to the use of simulation for instructional purposes in general 
and its use for training personnel in educational inquiry roles in 
particular. Special attention was then directed to a <ietermi nation of 
those areas wi<hin educational research, development, diffusion, and 
evaluation which may be amenable to a simulation format. 

Specific examination of the possible use of simulation to 
transmit those skills which were judged earlier to be most important 
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and in shortest supply (from Chapter 2) led to the following observations:' 

1. The most needed research skills are also those which seem 
least susceptible to simulation, reFquire the greatest amount of work 
to develop relevant simulation materials i and currently have no 
simulation materials available. 

2. In both development and diffusion , the most needed skills 
seem more susceptible to simulation and developing relevant simulations 
entails a more moderate amount of workj again, however, no simulation 
materials are currently available. 

3. The evaluation skills were judged as quite susceptible to 
presentation by simulation and developing relevant simulations requires 
only a moderate amount of work; in this category, simulation materials 

are available and are currently in use. '* ' \ 

Chapter 5 concluded with a discussion of developmental and 
administrative issues which must be taken into account in considering 
the uses of simulation for research training.; 

Recommendati ons 

On the basis of the summary above and the more detailed presen- 
tations in the previous chapters, the following recommendations seem 
in order. ^ These do not constitute a panacea for all the problems of 
ttie educational research community; such was not intende'd. Rather this 
was viewed as a pilot effort to explore the training needs of educational 
researchers and research-related personnel, to suggest possible methods 

— — « - - ■ — ' ' ' ■ « 

^Many of the- recommendations in this section are , incorporated in 
the oroposal for further Task Force activity during 1970-71, which appears 
in Appendix N. In some cases, that document provides a more detailed 
statement of specific tasks to be accomplished and objectives to be met. 



for improving the training of such personnel, and to recommend new or 
confinuing studies where appropriate, 

1. ""the classification^scheme developed so far in the course of this 

project represents substantial progress in describing the skills 
needed by educational research personnel. Nonetheless it is incom- 
plete. It needs refinement in some areas*, extension and expansion 
in others. Specifically: 

(a) Greater attention must be devoted to activities involving^ 
historical and philosophical inquiry activities which were 
neglected in aarlier work. ^ . . 

(b) A way must /e found to get at the substance of educational 
deveiopme/t and diffusion activities. (These activities are 
inadequately described at present; consequently the requisite 
skills^and knowledge* are ill-defined,) 

(c) Partly to achieve these goals, further testing of the classifi- 
cation system should involve a wider and probably a larger 
sampled in particular, the sample should include greater 
numbers of individuajs engaged in historical and philosophical 
iQquiry. 

(d> In addition, validation of the classification' system should 
utilize a task analysis approach and other techniques to 
identify skills and knowledge omitted ^rom the present lists 
and to make possible the description 6f major areas of activity 
which may have been overlooked in the present system. 

2. It is not known to what -extent personnel now involved in educational 

/ 

research, development, diffusion, and evaluation (RDDE) actually 
• possess the skills and knowledge which are regarded as necessary for 
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the successful performance of research-related tasks. Siinilarly, 

I 

it is fipt known to what extent participants In current research 
training programs are acquiring the essential skills and knowledge. 
It is therefore recommended that the foUowang activities be under- 
jjaken as steps toward prc^viding this information: 
(a) Devise a means of ass.essing the level of competence of RDDE 

personnel in critical s4<ills and knowledge- 
lb) 'On th)s basis of careful, in-depth examination of program 

content, assess the degree to which existing training programs 

specifically under Title IV provide the requisite skills 

-and knowledge. ' ' 

(c) Examine products (graduates) of individual r^seai^ch training 

programs in terms of career-indices such as research involvement 

and productivity ^mTrelate these, to differential patterns 

within individual programs. 

/ 

Since passage of the Elementary- and Secp|idary Education Act of 1965, 
it has been widely held that there will Le (or is) a great demand 
for RDDE personnel -'- a demand which alnjost certainly cannot be 
met by existing training programs. The Task Force cannot say with 
certainty whether such a situation now exists in the educational 
research community as a whole. On the basis of AERA employment 
' service data for 1968, 1969 and 1970, however, i.t appears that there 
is not presently a serious undersupply of personnel in research 
and research-related fields. On the contrary;, the number of positions 
open has declined over the three years studied, while the nutnber of 
applicants hds remained constant.. It is therefore reconmended that; , 
(a) Considerable effort should be devoted to determini|ig the 



nu!rtber_3 of RODE, positions which will exist over the next 
3 to 6 years anp^ the numbers of RDUE personnel who will be 
available to fill them, 
(b) As a part of that effort, an attempt should be made to 
determine the functional orientation of positions and 
available personnel, with particular attention to educational 
development and diffusion, 

4. The accomplishment of the tasks included under 1 , 2 and 3 above 
should establish the basis frjm which specific, substantive recom- 
mendations may be made concerning an optimal approach to research 
training. In the interim, however, it is still possible to suggest 
ways in which current training efforts might be improved. To that 
end the following statements and recommendations are offered for 
consideration: 

• (a) Since there is too much to communicate to graduate research 
trainees in the amount of time normally spent in a training 
program,^ greater attention should be given to recruitment. 
If trainees are re^/uited who already possess considerable 
knowledge and skill relatihg to the discipline, then more 
time may be spent within the training programs on those compe- 
tencies which are peculiar to research, development, dr ^ 
evaluatiori. 



This is true even if ofie accepts the earlier premise that not 
aji educational researchers need'^ in-depth training in all research 
specialities, Eveji the few essential skills mentioned and knowledge 
of a relevant discip'^^lne represent more content than can be co'mmuni cated 
to a train'ee in the, time non^ially allotted to graduate training. 
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(b) Outside of graduate programs, those involved in research 
training must at present depend on ancillary strategies for 
teaching many of the essentials/ Specifically, AERA should 
initiate efforts to provide short-term learning opportunities -- 
such as workshops, institutes, and self-contained, exportable, 
programmed materials which have promise of reaching broad 
audiences and^/providi ng training in some skills and knowledge 
now in short supply. 

(c) Given the adaptaB'i li ty of many of the development and diffusion 
skills to training through simulation, and the paucity of 
materials for such training, it is ufged that AERA initiate 
the development and testing of materials for appropriate skills 

w 

in these activities. 

(d) Many of the skills which have been described as critical for 
the successful conduct of educational inquiry are not now being 
effectively taught through formal coursework or tnrough tne 
ancillary strategies listed in (b) above. Rather, these skiRs^ 
(e.g., drawing conclusions, assessing goals, and reporting 

* results) are generally learned through experience, often under 
the tutelage of a senior researcher. For this reason, it is 
strongly urged that graduate training di rectors' make every 
attempt to provide meaningful practicum experiences preferably 
of an apprenticeship nature for their trainees, since it 



Thp present is stressed here in cognizance of th^ possibility 
that as more effective techniques are developedvwe may be able to do 
a more adequate job in fomial training and reduce dependence on some of 
these ancillary strategies. This*^simply remains to be seen. 
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seems to be the sole vehicle at present for inculcating certain 
critical skills and sensitivities, 

5. The recommendation that can be stated most unequivocally is that 
studies of the type proposed in Appendix N herein be continued. 
We know far toe little about many important variables and para- 
meters of training research and research-related personnel in 
education. It is ironic that researchers, who as a group spend 
a great deal of time generating data to test the effects of 
educational practices, have spent so little time generating infor- 
mation that would shed light on the efficacy of current practices 
in training educational researchers. Such information is badly 
needed before we can be assured that our training programs are 
based on a sound knowledge base and on systematic tryout and 
evaluation of alternative training modes. The present report 
represents only a tentative first step in this direction and 
further studies in the area are obviously needed. 
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